-
Ideas needed! How to develop an automatic web update alert by Python?
updated 4 years ago
2 Members
·
6
Posts
-
A friend needs an automatic web alert for her business. For example, if there are any new updates or modifications at out Activity page:
Then she can receive an alert of email notification automatically.
My rough idea is:
1. we can develop a web crawler by Python and get all the current web contents and find out the latest update date, modification date or activity date. Create a data set for the current contents.
2. Use Python scheduler to run the crawler each day or each hour to get the new web contents. Then we can compare the new latest update/modification dates etc with those from our history data.
3. If the new dates are later than historical ones, it means that the website is updated or modified. We can then send out an email alert and append the newly grabbed data into history data. If not, we do nothing but keep on checking the website by our crawler.
Is there any better method than this one? Please let me know, I am all ears, thank you!
Patrick, how do you think? You are an expert, please give me your suggestions. Thanks!
-
My 2 cents:
1. If it is possible use “push” instead of “pull”, for example push updates/alerts to emails with accurate changes;
2. If push is not available, try free webpage monitoring servies for example: https://visualping.io/
3. There are also desktop version of monitoring tools
4. Customize using Python tools: for example https://thp.io/2008/urlwatch/
5. Write your own code as what you’ve mentioned above
-
Thank you for your quick reply and valuable suggestions.
“1. If it is possible use “push” instead of “pull”, for example push updates/alerts to emails with accurate changes; “
I am not totally clear. You mean forward the new changes by email? It is a good idea but requires more work.
Let me check the webpage monitoring services available now, the Method 4 by Python tools is probably the best option because we can customize it.
-
I meant that if your friend’s website is owned by himself/herself, then the changes are already known in the database. It is easier to do a lot of alert either by scheduler, by db table trigger, or other mechanism without consistently checking on this webpage every minute or so with extra web traffic without knowing what exactly the change types are (forum/posts/rely, etc.)
If the website you are monitoring is owned by other people then the 1st option doesn’t really exist.
I am not totally clear. You mean forward the new changes by email? It is a good idea but requires more work.
-
-
I checked the urlwatch Python package, please see below, the fundamental idea is same as mine. Please see below…… we can just use this package since it is available and free.
———————————————-
Introduction
urlwatch monitors the output of webpages or arbitrary shell commands.
Every time you run urlwatch, it:
- retrieves the output and processes it
- compares it with the version retrieved the previous time (“diffing”)
- if it finds any differences, generates a summary “report” that can be displayed or sent via one or more methods, such as email
-
Log in to reply.