- Joined
- Mar 7, 2024
- Messages
- 11
- Reaction score
- 1
It has been 4 days, and finally a way around worked in the SSD cloud hosting option at HostMaria. So, I am going to emphasize on it.
The major issue is that the AI models don't work here, which is major hype in this world with ChatGPT, Midjourney, Gemini, and such. However, this is not the only use case for Python. It can be used in shell scripting, cryptocurrencies, scraping & crawling... Wait, I can make a simple web crawler that finds all the anchor tags on a website and shows them on my website?
So, I got back to researching. I knew the Python package 'beautifulsoup4' was the way to go; however, I got an even better option, Scrapy. It is designed for crawling around websites. Not only that, it has many other functionalities, but we'll stick to the core concept here.
Without any further delay, I 'sshed' into the cloud and ran:
Bbbut, it didn't work. It turns out Scrapy needs rust to be installed, as well as many other external dependencies such as libxml2 & libxslt packages, which I don't have access to. Hence, I decided to stick with 'beautifulsoup4' and 'requests' libraries.
And voila, no errors. After that, I one-click installed Laravel into the cloud and created a folder named in the framework's root directory (site-app). There, I created a file named crawl.py with the following contents:
The file gets a URL from input(), requests the URL, and, if it gets a valid response, finds all the anchor tags and collects their data into a list. Then it prints it out back to the web server.
From the Laravel side, the routes/web.php looks like this:
If the request arrives at the server with the 'url' parameter, it responds with the list of links the Python script has sent back. Here's a simple view I have set up with Bootstrap:
I know there might be some bugs, but you can mention them in the comments. Also, I have hosted the product here, so that you can check it out. There is even a GitHub repo for this, and I will update the repository if I ever touch it again.
Thank you for your time.
Prashanna Tamrakar
The major issue is that the AI models don't work here, which is major hype in this world with ChatGPT, Midjourney, Gemini, and such. However, this is not the only use case for Python. It can be used in shell scripting, cryptocurrencies, scraping & crawling... Wait, I can make a simple web crawler that finds all the anchor tags on a website and shows them on my website?
So, I got back to researching. I knew the Python package 'beautifulsoup4' was the way to go; however, I got an even better option, Scrapy. It is designed for crawling around websites. Not only that, it has many other functionalities, but we'll stick to the core concept here.
Without any further delay, I 'sshed' into the cloud and ran:
Code:
pip3 install --user scrapyBbbut, it didn't work. It turns out Scrapy needs rust to be installed, as well as many other external dependencies such as libxml2 & libxslt packages, which I don't have access to. Hence, I decided to stick with 'beautifulsoup4' and 'requests' libraries.
Code:
pip3 install --user beautifulsoup4
pip3 install --user requestsAnd voila, no errors. After that, I one-click installed Laravel into the cloud and created a folder named in the framework's root directory (site-app). There, I created a file named crawl.py with the following contents:
Code:
# site-app/python/crawl.py
import requests
from bs4 import BeautifulSoup
import json
url = input()
response = requests.get(url.split(' ')[0])
my_urls = []
if response.status_code == 200:
soup = BeautifulSoup(response.text, 'html.parser')
for link in soup.find_all('a'):
if link.get('href') == '#':
continue
my_urls.append({
'href': link.get('href') if not link.get('href').startswith('/') else url.rstrip('/') + link.get('href'),
'nofollow': link.get('rel') is not None and 'nofollow' in link.get('rel')}
)
print(json.dumps(my_urls, indent=4))
else:
print('{"error": "Something went wrong"}')The file gets a URL from input(), requests the URL, and, if it gets a valid response, finds all the anchor tags and collects their data into a list. Then it prints it out back to the web server.
From the Laravel side, the routes/web.php looks like this:
Code:
<?php
// routes/web.php
use Illuminate\Support\Facades\Route;
use Illuminate\Http\Request;
Route::get('/', function (Request $request) {
$links = [];
if (isset($request->url) && Str::of($request->url)
->test('/https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&\/\/=]*)/'))
{
$url = $request->url;
$output = Process::path(base_path('python'))->input($url)->run('python3 crawl.py')->output();
$links = json_decode($output, true);
if (isset($links['error'])) $links = [];
}
return view('crawler', data: [
'links' => $links,
]);
})->name('crawler');If the request arrives at the server with the 'url' parameter, it responds with the list of links the Python script has sent back. Here's a simple view I have set up with Bootstrap:
Code:
<!DOCTYPE HTML>
<html lang="en">
<head>
<title>Scraper</title>
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
<style>
.container {
margin-top: 50px;
}
div:has(input[type="checkbox"]:checked) + div [data-nofollow="true"] {
display: none;
}
</style>
</head>
<body>
<div class="container">
<div class="row">
<div class="col-md-6 col-md-offset-3">
<form action="{{ route('crawler') }}" method="GET">
<div class="form-group
{{ $errors->has('url') ? 'has-error' : '' }}">
<label for="url">
Enter URL
</label>
<input type="text" name="url" id="url" class="form-control" placeholder="Enter URL">
@if ($errors->has('url'))
<span class="help-block text-danger">
{{ $errors->first('url') }}
</span>
@endif
</div>
<div class="form-group text-center">
<button class="btn btn-success" type="submit">Crawl</button>
</div>
</form>
<label for="nofollow" class="inline">Hide Nofollow links</label>
<input type="checkbox" id="nofollow" name="nofollow" class="checkbox checkbox-inline" value="1">
</div>
<div class="col-md-6 col-md-offset-3">
@if (!empty(request('url')))
@forelse($links as $link)
<div class="alert alert-success" data-nofollow="{{ $link['nofollow'] }}">
<a href="{{ $link['href'] }}" style="word-wrap: break-word">{{ $link['href'] }}</a>
</div>
@empty
<div class="alert alert-danger">
No links found.
</div>
@endforelse
@endif
</div>
</div>
</div>
</body>
</html>I know there might be some bugs, but you can mention them in the comments. Also, I have hosted the product here, so that you can check it out. There is even a GitHub repo for this, and I will update the repository if I ever touch it again.
Thank you for your time.
Prashanna Tamrakar
