Show HN: Capillaries: Distributed data processing with Go and Cassandra https://ift.tt/pPu6h9z

Show HN: Capillaries: Distributed data processing with Go and Cassandra I started thinking about this approach after working on a large-scale project for a major financial company where our group developed a distributed in-house data processing solution. On a regular basis, it ingested a few gigabytes of financial data and, within a tight SLA time limit, produced a lot of enriched/aggregated/validated data for a number of customers. Sometimes, source data had errors, so operators with domain knowledge had to verify data validity at some checkpoints, immediately make corrections, and re-run parts of the workflow manually. The solution involved complex web service orchestration, custom database and was very demanding on the infrastructure availability. Capillaries is a built from scratch, open-source Go solution that does just that: ingests data and applies user-defined transforms - Go one-liner expressions, Python formulas, joins, aggregations, denormalization - using Cassandra for intermediate data storage and RabbitMQ for task scheduling. End users just have to provide: - source data in CSV files; - Capillaries script (JSON file) that defines the workflow and the transforms; - Python code that performs complex calculations (only if needed). The whole data processing pipeline can be split into separate runs that can be started independently and re-run by the user if needed. The goal is to build a platform that is tolerant to database and processing node failures, and allows users to focus on data transform logic and data quality control. “Getting started” Docker-based demo calculates ARK funds performance, using EOD holdings and transactions data acquired from public sources. There are also integration tests that use non-financial data. There is a test deploy tool that uses Openstack API for provisioning in the cloud. https://capillaries.io May 16, 2023 at 05:43AM

Comments

Post a Comment

Popular posts from this blog

Save Promotional Expense with Sure Shot Outbound Telemarketing Services Provider

Image
Advertising and promotions are essential for getting hold of new clients. In the competitive business world of today, it has become vital to expand one’s operation. If you stagnate, then your competitors will surpass you. However, advertisement-based promotions do not always guarantee a fixed return. You can spend copious sums of money on mass media advertising. But, it is not a given that your business will receive the anticipated response. And, even if you get the clients, they might not always be the most suitable for your business. Hence, it is important to exploit other alternatives. Partnering with a sure shot outbound telemarketing services provider can be a good start. Pinpoint Your Target with Outbound Telemarketing Services Provider Unlike an advertising campaign that relies on the quality of content and its appeal to a vast group of clients, outbound telemarketing campaigns can be pinpointed towards a particular client base. There are many benefits of employing t...
Read more

Monetize Fiber Services Better and Improve ROI with Top Telecom Billing Software

  Fiber internet connection has changed the way web services are utilized. With speeds in excess of 100 mbps, it is a revolutionary technology that is replacing the traditional copper wire connectivity. Subscribers are drawn towards this technology; however, not all telecom operators are able to monetize it in the best possible manner. There is a pressing need for next-gen telecom software solutions that come with backward compatibility. The Need for Backward Compatibility Fiber is a next-gen technology; however, it has to exist in a universe where copper wire and DSL connectivity is still a reality. Therefore, it needs to be backward compatible. Both the network technology and billing technology need to work with pre-existing infrastructure. So, it is vital that telcos have BSS and OSS software telecom along with relevant telecom billing software that work with older technologies. Flexible Monetization is Key Fiber offers a wide range of options for monetization. There ...
Read more

Find the best backlinks list and unlimited indexing process to know for improvement

https://sites.google.com/view/callcenterusa/home https://vcallglobal.bcz.com/ http://facebookhitlist.com/profiles/blogs/top-8-benefits-for-businesses-with-outsourced-call-center http://ampicillin500.com/Blog-Detail/Top-8-Benefits-For-Businesses-with-Outsourced-Call-Center-Services-1077 https://articlesforwebsite.com/top-8-benefits-for-businesses-with-outsourced-call-center-service/ https://traiborg.com/profile/vcallglobal https://issuu.com/ameliarobert/docs/some_of_identifying_the_best_telemarketing_service https://www.slideshare.net/ameliarobert/some-of-identifying-the-best-telemarketing-service-important https://slides.com/ameliarobert/some-of-identifying-the-best-telemarketing-service-important https://www.scribd.com/document/469110266/Some-of-Identifying-the-Best-Telemarketing-Service-Important https://visual.ly/community/Infographics/business/some-ide...
Read more