Only "parallel computing against machines" in Storm, less "parallel computing against each machine"?

Patrick Yu Source

I read a post about Storm below. What is the "task" in Storm parallelism

Both the question and the answer are very helpful for me, a beginner of Storm. However I have still some comments. It is said in the answer,

Running more than one task per executor does not increase the level of parallelism -- an >executor always has one thread that it uses for all of its tasks, which means that tasks >run serially on an executor.

Does it mean the "parallel computing" of Storm concentrate on "parallel machines". As Miguno mentioned, it gives us the flexibility to expand cluster machines. However the machine is becoming more and more CPU cores integrated, could it be more effeient if we use parallel computing against each machine in the cluster? (To some extent I felt the multi cores in each machine is also a "cluster"). There are also a couple of very helpful parallel programming libraries like AKKA, Disruptor, Killim and etc, which make us develop parallel programms conviniently.

I'm not sure if I misunderstand something about the Storm frame (or this kind of Could computing frame)?

Thanks,

multithreadingparallel-processingapache-storm

Answers

answered 5 years ago user2720864 #1

Bit confused what exactly you are asking. As mentioned in the same article that by increasing the Worker Process you can increase the number of Java process to run across the cluster for executing the tasks.

From the wiki it says

Config.TOPOLOGY_WORKERS: This sets the number of worker processes to use to execute the topology. For example, if you set this to 25, there will be 25 Java processes across the cluster executing all the tasks. If you had a combined 150 parallelism across all components in the topology, each worker process will have 6 tasks running within it as threads.

It can be done by doing something like

    Config conf = new Config();
    conf.setNumWorkers(25); 

By saying parallel computing if you intent to increase the number of Threads then you should be looking at the executors, by increasing the same you can spawn more threads per component. From the same article you mentioned it says

Note that as of Storm 0.8 the parallelism_hint parameter now specifies the initial number of executors (not tasks!) for that bolt.

e.g topologyBuilder.setBolt("green-bolt", new GreenBolt(), 2)

please correct me if I misunderstood what you are seeking

comments powered by Disqus