Innledning
Jeg jobber i et selskap hvor vi kjøre mange satsvise jobber behandling millioner av poster av data hver dag, og jeg har tenkt nylig om alle maskinene som sitter rundt hver eneste dag gjør ingenting for flere timer. Ville det ikke være bra hvis vi kunne bruke disse maskinene til å styrke behandlingskapasiteten til våre systemer? I dette settet av artikler jeg skal se på de potensielle fordelene ved å ansette et kontor rutenett med virtualiserte miljøer.
I del 1 ga jeg en oversikt over systemet og teknologiene jeg skal bruke så vel som diskutert noen av de mulige årsakene til at du ønsker å opprette et kontor rutenett.
Job Kontroll
Hvis du skal kjøre jobber så du kommer til å trenge noen måte å håndtere dem. Din jobb kontroll system (på jobben din server) må være veldig godt gjennomtenkt før du selv forsøker å kjøre et kontor rutenett. Så det første, hva er oppgaver for en jobb kontroll system:
- Del ut jobbene på forespørsel fra arbeiderne
- Fortell arbeidere hvilken type jobber å kjøre
- Spor jobber
- Sørg for at jobbene er kun kjørt en gang
- Gi jobbdata til arbeiderne, eller i det minste fortelle dem hvor de skal få det
Systemet må også være utvidbart, en løsning som fungerer for nå i et enkelt tilfelle kan utvides til å kjøre flere typer jobber som bedriften ser verdien i et rutenett løsning. For eksempel kan jobber gain prioriteringer, mer enn én type jobb kan foreligge (dvs. flere code baser), til slutt kan du også kjøre flere forskjellige arbeider maskiner som er optimalisert for hver type jobb (selv om det beveger seg bort fra den "generiske arbeideren 'idé). Prøver alltid å tenke på fremtiden når utvikle systemer, kan et kortsiktig syn føre til lengre sikt frustrasjon og økt utvikling tid.
Job Server
Vi kommer til å trenge et sted å kontrollere vår jobber fra, bør dette være den eneste systemet i nettet ditt som har en fast resource locator, være at en IP-adresse, vertsnavn, URL (bruk av interne DNS), osv. Dette er fordi behov for arbeiderne å vite hvor du skal lete etter jobber, arbeidere trenger å finne den jobben kontrollsystemet (ikke jobben kontrollsystemet finne arbeiderne).
Jobben serveren selv egentlig ikke har en komplisert oppgave (i et grunnleggende system hvertfall), det er behov for å lagre en liste over jobber, dele ut jobber, får resultater, og deretter lagre dem for senere henting. Hvordan disse delene (som "hand out jobber") er definert kan være svært grunnleggende. Senere kan vi utvide systemet til å inkludere et administrasjonsgrensesnitt for å legge til, redigere, slette, suspendere jobber, men dette er hinsides denne øvelsen.
Det er ingen grunn overhodet da at jobben din server ikke kunne være en virtuell maskin som kjører innenfor ditt viktigste behandlingen serveren forutsatt at det ikke tappe for mye ressurser fra det. Jobben server imidlertid ikke trenger høy tilgjengelighet, hvis det går ned på en fredag kveld du kommer til å miste en hel helg med behandling, potensielt koste deg et par uker igjen av behandlingstid (i forhold til dine viktigste behandlingen server alene) . Det kan være lurt å vurdere å sette jobben din server på et lass balansert miljø for høy tilgjengelighet.
Basic Setup
The basic setup for our job server will consist of what I'm calling one of my LiMP servers (that is Li nux, m ySql, P HP). The code running on the workers will actually work out what jobs it can run by interacting with with job control system databases. Later on we could create a web service and actually hand out jobs rather than having the workers do the hard work themselves, but for now we'll continue using the KISS principle (Keep it Simple, Stupid!).
So, lets create three mySQL tables to deal with jobs. These will be `jobs`, `jobRecords`, and `jobResults`.
Here I'm using SQL Buddy a great little alternative to phpMyAdmin just because its easier to install on centOS (for others see: 10 Great alternatives to phpMyAdmin )
This table consists of 5 simple fields,
- id: Uniquely identify the job
- name: Could be a client reference, or any number of other identifiers
- Status: You need to know where the job is at, eg
- 0: Not started
- 1: Picked up
- 2: Completed
- started_by: Who's started doing the job? This isn't entirely required but is a nice to have. I'd suggest tracking workers by their IP address on your network
- started_at: When did the worker start the job? By tracking jobs that have not completed within X amount of time we know we need to pick up the job once again and start processing by another worker. Workers could stop processing/go offline for any number of reasons, power failure, crash, network loss, etc.
It is easy how this table could be extended with a few additional fields to allow for statistics tracking, a finish time column to see how long the job took, a counter to see how many workers picked up the job (obviously this needs to tend to 1), job priority, the list can go on and on. In more complex job scenarios it would be possible to specify how much memory the worker would need access to (and therefore only use suitable workers), or even what type of worker would be required.
Lets add a few example jobs:
The next table again is quite simple to understand, these are our job records. They are linked to the main jobs table by a column `jobs_id`. The make up of this table very much depends on the data that you need to supply to your workers, lets make a very simple example where we have four columns:
- id: ID of the record
- name: Person's name
- address: Person's address
- jobs_id: The job ID that this record is linked to
The third and final table consists of a results table, it has much the same make up as our records table, and with the addition of some columns could be part of the records table:
- job_record_id: Link the result to the job table
- result: The result data
…and that's all you need for job control! (albeit at a very basic level) In my case I'm pointed to another table where my data to process was located, but this could just as easily been a file, parameters to run simulation code, you name it.
Selecting a job
Som nevnt tidligere, vil arbeiderne gjøre jobben vår ledelse for oss for nå, så alt vi trenger for virkelig å gjøre er å finne en jobb som trenger behandling og få informasjon. Hvordan ville vi gjøre dette? Vel plukke jobben vår utvalgskriterier og lete etter jobber, i SQL gjorde jeg følgende:
- Ta noen jobber som ikke er merket som komplett, men fra arbeideren vår og nullstille dem (innbytter __ME__ med en identifikator, ville enkleste være IP-adresse):
UPDATE `jobb` SET `status` = 0 HVOR `status` = 1 AND `started_by` = __ME__;
- Ved hjelp av vår jobb seleksjonskriterier, velger du en jobb og fortelle kontrollsystem som denne arbeideren har å gjøre med det:
UPDATE `jobb` SET `status` = 1, `started_by` = __ME__, `started_at` = NÅ () HVOR `status` = 0 eller
(`Status` = 1 AND `started_at`> DATE_SUB (NÅ (), INTERVALL X HOUR)) ORDER BY `id` ASC;
Ved å ta tak i jobber som ikke har returnert resultater i X tid vi sikre at alle jobber kjøres i tilfelle en arbeidstaker å krasje eller gå AWOL.
- Neste tak i jobbene detaljer etterfulgt av postene selv:
SELECT * FROM `jobb` HVOR `started_by` = __ME__ LIMIT 1;
SELECT * FROM `job_records` HVOR `id` = __JOBID__;
Ved ferdigstillelse av jobben vi sette inn våre resultat poster og mark jobben som fullført. Husk som jobber kan suspendere / gjenoppta når som helst la for noen robusthet i skriptet. Det kan være at oppgaven suspenderer halvveis gjennom oppdatering jobben kontrollsystem, så sjekker antall poster i en jobb og antall resultater lagres tilbake til jobben styresystem ville være et klokt trekk.
In addition, whilst this demonstrates how jobs can be selected and managed from an SQL-query frame you should really be abstracting your job control so that if you decide to switch to using a web service, a file based system, XML , or any other number of systems it will not affect the code above it.
Job Configuration
The next aspect to consider is job size and configuration. By playing with job configuration we can strike an excellent balance between speed, process replication, and reliability. Take a couple of scenarios:
- Jobs take 1 day each to run: This means that your workers need 15 days to process each job (remember 10% of the power for 2/3rds of the time). This is clearly not a wise configuration, your job size is way too big! It would take at least double the time to get a job processed should the initial worker go AWOL (time to pick up that it hasn't returned a result plus reprocessing time). In an ideal you'd have at least one full job easily cleared by the end of each long idle period, that way you keep the jobs ticking over and at worst case a job would take two days to process should the first go missing.
- Jobs take 1 minute to run: This means that your workers take about 15 minutes to run each job. Whilst this may initially seem ideal, you gain additional work processing during lunch time, coffee breaks, meetings, etc this scenario puts strain on other areas of your system and introduces its own problems. For example, firstly your setup/processing time ratio is going to go right down, therefore losing system efficiency. Your network is going to be constantly streaming job information to the various workers frustrating staff who are dong their day to day work. You're also going to put more strain on your job processing server as it has to dish out lots and lots of small pieces of work on a regular basis. Lastly, in this situation if your job server goes down you're going to create a huge back log of uncompleted work whereas bigger jobs could of continued processing blissfully unaware that the job server was experiencing difficulties.
In reality there will be no one ideal configuration for your grid setup, much depends on the available resources, types of job, job turnaround time requirements, network capability, and so on. However some guidelines would be:
- Size jobs so that each worker can get through at least 3-4 jobs in a period of 15 hours (the longest likely idle time period)
- Play with the job size so that setup time becomes fairly insignificant compared to the processing time (bearing in mind the above point).
- If a job doesn't complete in double the amount of time (maybe less) you expect it to complete it assume that its gone AWOL and start processing it with another worker. This means you may have to wait up to three times the normal length of a job for it to complete (possibly longer if the subsequent job fails). You may want to reduce this time, but be careful not to reduce it too much as you may start duplicating processing tasks on a regular basis.
- Jobs should be independent of outside requirements as much as possible. The job server, for example, should only be contacted at the start and end of every job.
- Don't saturate your network, this will have two negative effects, your daytime staff will find using the network frustrating and problems may be experienced with connections timing out a problem that will only get worse as you scale your grid.
- Ensure jobs can run on your workers. If jobs become too memory intensive or disk space intensive jobs will start aborting and the only thing you'll notice is a drop in number of jobs processed with no real reason why.
Submitting Results of a Job
When submitting the results of a job it is important to check that results have not been submitted by another worker, especially if the current worker has been dormant for some time.
When results are submitted ensure that the number of results matches the number of records within the job.
As stated previously, and can not be over emphasised, build fault tolerance into job retrieval and results submission. The workers can (and most likely will) go into suspend mode at the most inconvenient of times and this needs to be catered for. Also once again abstracting away your results submission will help cater for future changes to your job control system much easier to deal with.
Oppsummering
In this section we have looked at what a job control server needs to do and how to get a very basic system set up. We discussed how to retrieve a job from the control system and how best to configure jobs to get the most our of your office grid system. To finish, a paragraph or two on submitting results back to the job control server was presented.
- A job control server manages jobs and ensures that all work units are completed
- By abstracting your job select/results submission we can change the technology of the control server without much problems
- Configure your jobs to ensure that they are run quickly and efficiently without putting too much pressure on your network infrastructure, and without duplicating processing tasks on a regular basis.
- Ensure that you build fault tolerance and error checking into your routines, workers can suspend and resume and the most inconvenient of times. Remember to check if results have already been submitted by another worker.
Next time
In part 3 we'll create our virtual processing machine and set up our windows machines to become idle-time workers.