Pengenalan
I work in a company where we run many batch jobs processing millions of records of data each day and I've been thinking recently about all the machines that sit around each and every day doing nothing for several hours. Wouldn't it be good if we could use those machines to bolster the processing power of our systems? In this set of articles I'm going to look at the potential benefits of employing an office grid using virtualised environments.
In Part 1 I gave an overview of the system and technologies I will be using as well as discussed some of the potential reasons why you would want to create an office grid.
Job Control
If you're going to be running jobs then you're going to need some way to manage them. Your job control system (on your job server) needs to be really well thought out before even attempting to run an office grid. So firstly, what are the tasks for a job control system:
- Hand out jobs upon request from workers
- Tell workers what type of jobs to run
- Track jobs
- Ensure that jobs are only run once
- Provide job data to workers, or at least tell them where to get it
The system also needs to be extensible, a solution that works for now in a single case may be extended to run several types of jobs as the business sees the worth in a grid solution. For example, jobs may gain priorities, more than one job type may exist (ie several code bases), eventually you may even run several different worker machines that are optimised for each type of job (although that does move away from the 'generic worker' idea). Always try to think about the future when developing systems, a short term vision can lead to longer term frustration and increased development time.
Job Server
We're going to need somewhere to control our jobs from, this should be the only system in your grid that has a fixed resource locator, be that an IP address, host name, URL (using internal DNS), etc. This is because the workers need to know where to look for jobs, workers need to find the job control system (not the job control system find the workers).
The job server itself doesn't really have a complicated task (in a basic system anyhow), it needs to store a list of jobs, hand out jobs, receive results, and subsequently store them for later retrieval. Bagaimana bagian-bagian ini ('menyerahkan pekerjaan' seperti) didefinisikan bisa sangat dasar. Later on we can extend the system to include an administration interface to add, edit, delete, suspend jobs but this is beyond this exercise.
There is no reason whatsoever then that your job server could not be a virtual machine running within your main processing server provided it doesn't drain too many resources from it. The job server however does need high availability, if it goes down on a Friday evening you're going to lose a whole weekend of processing, potentially costing you a couple of weeks worth of processing time (when compared to your main processing server alone). You may want to consider putting your job server on a load balanced environment for high availability.
Basic Setup
The basic setup for our job server will consist of what I'm calling one of my LiMP servers (that is Li nux, m ySql, P HP). The code running on the workers will actually work out what jobs it can run by interacting with with job control system databases. Later on we could create a web service and actually hand out jobs rather than having the workers do the hard work themselves, but for now we'll continue using the KISS principle (Keep it Simple, Stupid!).
So, lets create three mySQL tables to deal with jobs. These will be `jobs`, `jobRecords`, and `jobResults`.
Here I'm using SQL Buddy a great little alternative to phpMyAdmin just because its easier to install on centOS (for others see: 10 Great alternatives to phpMyAdmin )
This table consists of 5 simple fields,
- id: Uniquely identify the job
- name: Could be a client reference, or any number of other identifiers
- Status: You need to know where the job is at, eg
- 0: Not started
- 1: Picked up
- 2: Completed
- started_by: Who's started doing the job? This isn't entirely required but is a nice to have. I'd suggest tracking workers by their IP address on your network
- started_at: When did the worker start the job? By tracking jobs that have not completed within X amount of time we know we need to pick up the job once again and start processing by another worker. Workers could stop processing/go offline for any number of reasons, power failure, crash, network loss, etc.
It is easy how this table could be extended with a few additional fields to allow for statistics tracking, a finish time column to see how long the job took, a counter to see how many workers picked up the job (obviously this needs to tend to 1), job priority, the list can go on and on. In more complex job scenarios it would be possible to specify how much memory the worker would need access to (and therefore only use suitable workers), or even what type of worker would be required.
Lets add a few example jobs:
The next table again is quite simple to understand, these are our job records. They are linked to the main jobs table by a column `jobs_id`. The make up of this table very much depends on the data that you need to supply to your workers, lets make a very simple example where we have four columns:
- id: ID of the record
- name: Person's name
- address: Person's address
- jobs_id: The job ID that this record is linked to
The third and final table consists of a results table, it has much the same make up as our records table, and with the addition of some columns could be part of the records table:
- job_record_id: Link the result to the job table
- result: The result data
…and that's all you need for job control! (albeit at a very basic level) In my case I'm pointed to another table where my data to process was located, but this could just as easily been a file, parameters to run simulation code, you name it.
Selecting a job
As stated previously, the workers will do our job management for us for now, so all we need to really do is find a job that needs processing and get the information. How would we do this? Well pick our job selection criteria and look for jobs, in SQL I did the following:
- Take any jobs that are not marked as complete but from our worker and reset them (substitute __ME__ with an identifier, easiest would be IP address):
UPDATE `jobs` SET `status` = 0 WHERE `status` = 1 AND `started_by` = __ME__;
- Using our job selection criteria, select a job and tell the control system that this worker is dealing with it:
UPDATE `jobs` SET `status` = 1, `started_by` = __ME__, `started_at` = NOW() WHERE `status` = 0 OR
(`status` = 1 AND `started_at` > DATE_SUB(NOW(), INTERVAL X HOUR)) ORDER BY `id` ASC;
By grabbing jobs that haven't returned results in X amount of time we ensure that all jobs are run in the event of a worker crashing or going AWOL.
- Next grab the jobs details followed by the records themselves:
SELECT * FROM `jobs` WHERE `started_by` = __ME__ LIMIT 1;
SELECT * FROM `job_records` WHERE `id` = __JOBID__;
Upon completion of the job we insert our result records and mark the job as complete. Remember as jobs can suspend/resume at any time allow for some robustness in your script. It might be that the task suspends half way through updating the job control system, so checking the number of records in a job and the number of results saved back to the job control system would be a wise move.
In addition, whilst this demonstrates how jobs can be selected and managed from an SQL-query frame you should really be abstracting your job control so that if you decide to switch to using a web service, a file based system, XML , or any other number of systems it will not affect the code above it.
Job Configuration
The next aspect to consider is job size and configuration. By playing with job configuration we can strike an excellent balance between speed, process replication, and reliability. Take a couple of scenarios:
- Jobs take 1 day each to run: This means that your workers need 15 days to process each job (remember 10% of the power for 2/3rds of the time). This is clearly not a wise configuration, your job size is way too big! It would take at least double the time to get a job processed should the initial worker go AWOL (time to pick up that it hasn't returned a result plus reprocessing time). In an ideal you'd have at least one full job easily cleared by the end of each long idle period, that way you keep the jobs ticking over and at worst case a job would take two days to process should the first go missing.
- Jobs take 1 minute to run: This means that your workers take about 15 minutes to run each job. Whilst this may initially seem ideal, you gain additional work processing during lunch time, coffee breaks, meetings, etc this scenario puts strain on other areas of your system and introduces its own problems. For example, firstly your setup/processing time ratio is going to go right down, therefore losing system efficiency. Your network is going to be constantly streaming job information to the various workers frustrating staff who are dong their day to day work. You're also going to put more strain on your job processing server as it has to dish out lots and lots of small pieces of work on a regular basis. Lastly, in this situation if your job server goes down you're going to create a huge back log of uncompleted work whereas bigger jobs could of continued processing blissfully unaware that the job server was experiencing difficulties.
Pada kenyataannya tidak akan ada konfigurasi yang ideal satu untuk konfigurasi jaringan Anda, banyak tergantung pada sumber daya yang tersedia, jenis pekerjaan, persyaratan pekerjaan perputaran waktu, kemampuan jaringan, dan sebagainya. Namun beberapa pedoman akan:
- Ukuran pekerjaan sehingga setiap pekerja bisa mendapatkan pekerjaan melalui setidaknya 3-4 dalam periode 15 jam (periode waktu terlama kemungkinan idle)
- Bermain dengan ukuran pekerjaan sehingga waktu setup menjadi cukup signifikan dibandingkan dengan waktu proses (mengingat titik di atas).
- Jika pekerjaan tidak lengkap dalam dua kali lipat jumlah waktu (mungkin kurang) Anda harapkan untuk menyelesaikannya berasumsi bahwa AWOL yang hilang dan mulai memproses dengan pekerja lain. Ini berarti Anda mungkin harus menunggu hingga tiga kali normal panjang pekerjaan sampai selesai (mungkin lebih lama jika pekerjaan berikutnya gagal). Anda mungkin ingin mengurangi waktu ini, tapi hati-hati untuk tidak mengurangi terlalu banyak karena Anda dapat mulai duplikasi tugas pengolahan secara teratur.
- Pekerjaan harus independen dari persyaratan luar sebanyak mungkin. Server pekerjaan, misalnya, hanya harus dihubungi pada awal dan akhir setiap pekerjaan.
- Tidak jenuh jaringan Anda, ini akan memiliki dua efek negatif, staf siang hari Anda akan menemukan menggunakan jaringan frustasi dan masalah mungkin dialami dengan koneksi waktu keluar masalah yang hanya akan bertambah buruk sebagai Anda skala grid Anda.
- Pastikan pekerjaan dapat berjalan pada pekerja Anda. Jika pekerjaan menjadi terlalu memori pekerjaan ruang intensif atau disk yang intensif akan mulai membatalkan dan satu-satunya hal yang akan Anda perhatikan adalah penurunan jumlah pekerjaan yang diproses dengan tidak ada alasan sebenarnya mengapa.
Menyerahkan Hasil Pekerjaan
Ketika mengirimkan hasil pekerjaan penting untuk memeriksa bahwa hasilnya belum diserahkan oleh pekerja yang lain, terutama jika pekerja saat ini telah terbengkalai selama beberapa waktu.
Ketika hasilnya diserahkan memastikan bahwa jumlah hasil sesuai dengan jumlah record dalam pekerjaan.
Seperti yang dinyatakan sebelumnya, dan tidak bisa terlalu ditekankan, membangun toleransi kesalahan dalam pengambilan dan penyerahan hasil pekerjaan. Para pekerja dapat (dan kemungkinan besar akan) masuk ke mode Suspend pada yang paling nyaman kali dan ini harus dipenuhi. Juga sekali lagi pergi penyerahan abstrak hasil Anda akan membantu memenuhi untuk perubahan masa depan untuk sistem kontrol pekerjaan Anda jauh lebih mudah untuk menangani.
Ringkasan
Dalam section ini kita telah melihat apa server kontrol pekerjaan perlu dilakukan dan bagaimana untuk mendapatkan sistem yang sangat dasar mengatur. Kami membahas bagaimana untuk mengambil pekerjaan dari sistem kontrol dan bagaimana cara terbaik untuk mengkonfigurasi pekerjaan untuk mendapatkan yang paling kami sistem grid kantor Anda. Untuk menyelesaikan, satu atau dua paragraf tentang cara mengirimkan hasilnya kembali ke server kontrol pekerjaan disajikan.
- Sebuah server kontrol pekerjaan mengelola pekerjaan dan memastikan bahwa semua unit kerja selesai
- By abstracting your job select/results submission we can change the technology of the control server without much problems
- Configure your jobs to ensure that they are run quickly and efficiently without putting too much pressure on your network infrastructure, and without duplicating processing tasks on a regular basis.
- Ensure that you build fault tolerance and error checking into your routines, workers can suspend and resume and the most inconvenient of times. Remember to check if results have already been submitted by another worker.
Next time
In part 3 we'll create our virtual processing machine and set up our windows machines to become idle-time workers.
Komentar (4) 
Artikel , PHP , Web Programming , Zend Framework | 
kerangka kerja , ini , PHP , Router , Rute , Zend , Zend Framework , Zend_Navigation , Zend_Router 
