melezhik @ melezhik @programming.dev

Posts

11

Comments

16

Joined

2 yr. ago

TLTR;

Sparky is a distributed jobs framework that allows orchestration of remote tasks on cluster of nodes. It’s simple to set up and easy to use. This post is a brief overview of Sparky architecture design.

Sparky targeted audience is:

cloud providers to manage underlying multiple hosts infrastructure

data scientists to process data in distributed manner (aka data pipelines)

software engineers and devops doing any tasks with distributed nature in mind

You may try out https://github.com/melezhik/sparky which is a local / remote task runner with nice front end and scripts could be written on many languages

Yep. Fancy devs watching me coding some Rakulang in nano 😂

Thanks, will take a look

nano is the best (imho) for up to medium size files. It’s preinstalled in most Linux boxes , it’s simple and flexible enough, takes a minimal amount of time to learn basic for keys and then use them all the time

Not generator, validator. It validates configuration files . Ansible is not flexible in comparison with Sparrow, you'd need to write more boilerplate code to do the same ... Also core ansible modules search is limited by "one line" mode, thus it does not allow to search for example within nested structures, like if we want something in between or in nested blocks, or search for sequences, like when we want to search a sequence of strings, a,b,c,d etc, Sparrow does allow al thatl as it has ranges/sequential/SLN search by design. Sparrow allows to generate check rules in runtime as well, Ansible can't

fair enough, however the intention is to show how one could create rules on Sparrow/Raku, not to show rules ... Maybe I should have mentioned that ...

for example this is more interesting example evaluation of net.ipv4.tcp_synack_retries"

    
regexp: ^^ "net.ipv4.tcp_synack_retries" \s* "=" \s* (\d+) \s* $$

generator: <<RAKU
!raku
if matched().elems {
  my $v = capture()[];
  say "note: net.ipv4.tcp_synack_retries={$v}";
  if $v >= 3 && $v <= 5 {
     say "assert: 1 net.ipv4.tcp_synack_retries in [3..5] range"
  } else {
     say "assert: 0 net.ipv4.tcp_synack_retries in [3..5] range"
  }
} else {
  say "note: net.ipv4.tcp_synack_retries setting not found"
}
RAKU

  

sorry, could you please elaborate on "shouldn’t copy" ? thanks

you are seemed to have edited your initial reply - "it should be sysctl.conf not syslog.conf " - anyway thanks for that, now it's fixed, this was just overlook typo

Yep. Like said - "We talk about use of Bash for simple enough tasks ... where every primitive language or DSL is ok", so Bash does not suck in general and I myself use it a lot in proper domains, but I just do not use it for tasks / domains with complexity ( in all senses, including, but not limited to team work ) growing over time ...

We are not taking about use of Bash in dev vs use Bash in production. This is imho incorrect question that skirts around the real problem in software development. We talk about use of Bash for simple enough tasks where code is rarely changed ( if not written once and thrown away ) and where every primitive language or DSL is ok, where when it comes to building of medium or complex size software systems where decomposition, complex data structures support, unit tests, error handling, concurrency, etc is a big of a deal - Bash really sucks because it does not allow one to deal with scaling challenges, by scaling I mean where you need rapidly change huge code base according changes of requirements and still maintain good quality of entire code. Bash is just not designed for that.

Let me generalize that - yaml pipelines are terrible 😀

Ok. Huge part of building microservices framework is infrastructure automation - like setup nginx load balancing in runtime, build and deploy apps from source code, configuring services, tcp ports, health checks, horizontal scaling (adding new worker nodes), setup logging and monitoring, etc, also this needs to be propagated to all cluster nodes, I am not going to do this from the scratch - Sparky is alike (rough comparison though) ansible but with UI and programmable on Raku, so as Sparky has already addressed the mentioned tasks, it's logical for me to carry on with it. If we take Sparky out of equation, Raku by itself is reach and super flexible language to automate infrastructure, I don't see why can't I use it for that ...

The plan is to build the entire system on top of Sparky which is written on Raku and extended by Raku

What is an advantage of using yaml based dsl over regular programming language?

Also Sparrow itself is already a DSL, so task-run() functions is all one needs to build a pipeline

my 2 cents here, though I don't understand all the context, you might take a look at sparky - which is lightweight task runner with web console, so you may throw a bunch of jobs into it to do all the "bootstrapping" so that you may later repeat the same if required on any fresh environment ...