javascript

Question

328 votes

3 answers

javascript - Better way to schedule cron jobs based on job orders from php script

Get the solution ↓↓↓

So I wrote simple video creator script in NodeJS.

It's running on scheduled cron job.

I have a panel written in PHP, user enter details and clicks "Submit new Video Job" Button. This new job is saving to DB with details, jobId and status="waiting" data.

PHP API is responsible for returning 1 status at a time, checks status="waiting" limits query to 1 then returns data with jobID when asked

Video Creation Script requests every x seconds to that API asks for new job is available.

It has 5 tasks.

available=true.

Check if new job order available (With GET Request in every 20 seconds), if has new job; available=false
Get details (name, picture url, etc.)
Create video with details.
Upload Video to FTP
Post data to API to update details. And Mark that job as "done"

available=true;

These tasks are async so everytask has to be wait previous task to be done.

Right now, get or post requesting api if new job available in every 20 seconds (Time doesnt mattter) seems bad way to me.

So any way / package / system to accomplish this behavior?

Code Example:

const cron = require('node-cron');
let available=true;

var scheduler = cron.schedule(
    '*/20 * * * * *',
    () => {
        if (available) {
            makevideo();
        }
    },
    {
        scheduled: false,
        timezone: 'Europe/Istanbul',
    }
);


let makevideo = async () => {
    available = false;
    let {data} = await axios.get(
        'https://api/checkJob'
    );
    if (data == 0) {
        console.log('No Job');
        available = true;
    } else {
        let jobid = data.id;
    await createvideo();
    await sendToFTP();
    
        await axios.post('https://api/saveJob', {
            id: jobid,
            videoPath: 'somevideopath',
        });
        available = true;
    }
};
scheduler.start();

Undefined asked

2021-12-1

Write your answer

Share solution ↓

Additional Information:

Date the issue was resolved:

2021-12-1

Link To Source
Link To Answer People are also looking for solutions of the problem: the browser (or proxy) sent a request that this server could not understand.

Didn't find the answer?

Our community is visited by hundreds of web development professionals every day. Ask your question and get a quick answer for free.

Ask a Question

Write quick answer

Do you know the answer to this question? Write a quick response to it. With your help, we will make our community stronger.

score 374 · Answer 1

Answer

Solution:

RabbitMQ is also a good queueing system.

Why ?

It's really well documented (examples for many languages including javascript & php).

Tutorials are simple while they're exposing real use cases.

It has a REST API.

It ships with a monitoring UI.

How to use it to solve your problem ?

On the job producer side : send messages (jobs) to a queue by following tutorial 1

To consume jobs with your nodejs process : see RabbitMQ's tutorial 2

Other suggestions :

Use a prefetch value of 1 and publisher confirms so you can ensure that an instance of consumer will not receive messages while there's a job running.

Roadmap for a quick prototype : tutorial 1... then tutorial 2 x). After sending and receiving messages you can explore the options you can set on queues and messages

Nodejs package : http://www.squaremobius.net/amqp.node/

PHP package : https://github.com/php-amqplib/php-amqplib

score 785 · Answer 2

Answer

Solution:

While it is possible to use the database as a queue, it is commonly known as an anti-pattern (next to using the database for logging), and as you are looking for:

So any way / package / system to accomplish this behavior?

I use the free-form of your question thanks to the placed bounty to suggest: .

Beanstalk is a simple, fast work queue.

Its interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously.

It has client libraries in the languages you mention in your question (and many more), is easy to develop with and to run in production.

score 861 · Answer 3

Answer

Solution:

What you are doing in a very standard system design paradigm, done with Apache Kafka or any queue based implementation(ex, RabbitMQ). You can check out about Kafka/rabbitmq but basically Not going into details:

There is a central Queue.
When user submits a job the job gets added to the Queue.
The video processor runs indefinitely subscribing to the queue.

You can go ahead and look up : https://www.gentlydownthe.stream/ and you will recognize the similarities on what you are doing.

Here you don't need topoll yourself, you need to subscribe to an event and the other things will be managed by the respective queues.

javascript - Better way to schedule cron jobs based on job orders from php script

Answer

Solution:

RabbitMQ is also a good queueing system.

Answer

Solution:

Answer

Solution:

Share solution ↓

Additional Information:

Didn't find the answer?

Similar questions

Write quick answer

About the technologies asked in this question

PHP

HTML

Welcome to programmierfrage.com

Get answers to specific questions

Help Others Solve Their Issues