queue system using mod_perl2/apache2
[prev] [thread] [next] [Date index for 2005/05/23]
From: Cahill, Earl
Subject: queue system using mod_perl2/apache2
Date: 21:27 on 23 May 2005
Subject: queue system using mod_perl2/apache2
Date: 21:27 on 23 May 2005
This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C55FDE.3F9382E8 Content-Type: text/plain I am wondering a couple things First, are there any good, open source, perl-based queue systems out there? I think I would like a stable, centralized daemon, which likely rules out many systems. Second, is anyone interesting in working on a queue system with mod_perl2/apache2 as it's basis, using the mod_echo, arbitrary protocol stuff? Basically, you would register servers and jobs with the daemon, and then the daemon would give jobs out, perhaps wait for them to finish and then return appropriate output. The system would also handle job dependencies. I would like to be pretty open, and would hope to easily handle some basic job types, like 1. for each row returned from a db query, do something with it 2. watch a filesystem for changes to a directory or file and act accordingly 3. be fast enough to serve web hits through it, like maybe 1/100 second system overhead 4. have a flat file and do something for each line in it 5. be able to rather arbitrarily add little jobs, as in a web crawler running in parallel I think using mod_perl2/apache2 would be ideal. Could use apache2 to give jobs to non-mod_perl processes if desired. I think that if apache2 could manage forking, children and the like that would save many headaches. I think I even have a name, PDQ, which could stand for Perl Dynamic Queue, or yeah, Pretty Darn Quick. Feed back welcome. Thanks, Earl ------_=_NextPart_001_01C55FDE.3F9382E8 Content-Type: text/html Content-Transfer-Encoding: quoted-printable <html xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns=3D"http://www.w3.org/TR/REC-html40">; <head> <META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; = charset=3DUS-ASCII"> <meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)"> <style> <!-- /* Font Definitions */ @font-face {font-family:"MS Mincho"; panose-1:2 2 6 9 4 2 5 8 3 4;} @font-face {font-family:"\@MS Mincho"; panose-1:0 0 0 0 0 0 0 0 0 0;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman";} a:link, span.MsoHyperlink {color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {color:purple; text-decoration:underline;} span.EmailStyle17 {mso-style-type:personal-compose; font-family:Arial; color:windowtext;} @page Section1 {size:8.5in 11.0in; margin:1.0in 1.25in 1.0in 1.25in;} div.Section1 {page:Section1;} /* List Definitions */ @list l0 {mso-list-id:1287272221; mso-list-type:hybrid; mso-list-template-ids:87051360 67698703 67698713 67698715 67698703 = 67698713 67698715 67698703 67698713 67698715;} ol {margin-bottom:0in;} ul {margin-bottom:0in;} --> </style> </head> <body lang=3DEN-US link=3Dblue vlink=3Dpurple> <div class=3DSection1> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'>I am wondering a couple = things<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'><o:p> </o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'>First, are there any good, open source, perl-based = queue systems out there? I think I would like a stable, centralized = daemon, which likely rules out many systems.<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'><o:p> </o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'>Second, is anyone interesting in working on a queue = system with mod_perl2/apache2 as it's basis, using the mod_echo, arbitrary protocol stuff?<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'><o:p> </o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'>Basically, you would register servers and jobs with = the daemon, and then the daemon would give jobs out, perhaps wait for them = to finish and then return appropriate output. The system would also = handle job dependencies.<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'>I would like to be pretty open, and would hope to = easily handle some basic job types, like <o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'><o:p> </o:p></span></font></p> <ol style=3D'margin-top:0in' start=3D1 type=3D1> <li class=3DMsoNormal style=3D'mso-list:l0 level1 lfo1'><font size=3D2 = face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial'>for each row returned = from a db query, do something with it<o:p></o:p></span></font></li> <li class=3DMsoNormal style=3D'mso-list:l0 level1 lfo1'><font size=3D2 = face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial'>watch a filesystem = for changes to a directory or file and act = accordingly<o:p></o:p></span></font></li> <li class=3DMsoNormal style=3D'mso-list:l0 level1 lfo1'><font size=3D2 = face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial'>be fast enough to = serve web hits through it, like maybe 1/100 second system = overhead<o:p></o:p></span></font></li> <li class=3DMsoNormal style=3D'mso-list:l0 level1 lfo1'><font size=3D2 = face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial'>have a flat file and = do something for each line in it<o:p></o:p></span></font></li> <li class=3DMsoNormal style=3D'mso-list:l0 level1 lfo1'><font size=3D2 = face=3DArial><span style=3D'font-size:10.0pt;font-family:Arial'>be able to rather = arbitrarily add little jobs, as in a web crawler running in = parallel<o:p></o:p></span></font></li> </ol> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'><o:p> </o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'>I think using mod_perl2/apache2 would be ideal. = Could use apache2 to give jobs to non-mod_perl processes if desired. I = think that if apache2 could manage forking, children and the like that would save = many headaches.<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'><o:p> </o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'>I think I even have a name, PDQ, which could stand = for Perl Dynamic Queue, or yeah, Pretty Darn Quick.<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'><o:p> </o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'>Feed back welcome.<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'><o:p> </o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'>Thanks,<o:p></o:p></span></font></p> <p class=3DMsoNormal><font size=3D2 face=3DArial><span = style=3D'font-size:10.0pt; font-family:Arial'>Earl<o:p></o:p></span></font></p> </div> </body> </html> ------_=_NextPart_001_01C55FDE.3F9382E8--
 |
queue system using mod_perl2/apache2
|
|
 |
Re: queue system using mod_perl2/apache2
|