Archives
Trending
Support
Login
clear text
XML
Django
JavaScript
MATLAB
C
C++
C#
Python
SQL
Shell
Bash
Markdown
YAML
JSON
HTML
CSS
PHP
Java
Ruby
Go
Rust
Swift
Kotlin
Arduino
TypeScript
Perl
Autohotkey
Lua
SQF
R
Scala
Haskell
Groovy
Dart
Clojure
VB.NET
Objective-C
PowerShell
Bash
CoffeeScript
Verilog
#!/bin/bash #gets user input url="$1" #create uniq file name for this search searched="$(date +%s).lst" touch $searched echo "Saving to $searched" function crawl(){ url="$1" #s will be used to check if url has already been searched s=0 #check searched file for current url grep "$url" $searched && s=1 || s=0 #if current url has been search skip it if [[ $s == 0 ]]; then #add current url to searched list echo "$url" >> $searched echo "Crawling ${url}..." #get all urls on page lynx --dump "$url"|\ sed 's/http/\nhttp/g'|\ grep -e "^http:" -e "^https:"|\ sed 's/%3A%2F%2F/:\/\//g'|\ sort -u| while read line do #crawl through each url crawl "$line" done else #reset searched check s=0 fi } #calls crawl function crawl "$url"
Mark as private (unlisted)
for 30 minutes
for 6 hours
for 1 day
for 1 week
for 1 month
for 1 year
everlasting (like CCCP)
