Btukfyl

I would love some assistance or help around an issue i'm currently having.
I'm working on a little python scanner as a project.
The libraries im current importing are:

requests

BeautifulSoup

re

tld

The exact issue is regarding 'scope' of the scanner.
I'd like to pass a URL to the code and have the scanner grab all the anchor tags from the page, but only the ones relevant to the base URL, ignoring out of scope links and also subdomains.

Here is my current code, i'm by no means a programmer, so please excuse sloppy inefficient code.

import requests

from bs4 import BeautifulSoup

import re

from tld import get_tld, get_fld



#This Grabs the URL

print("Please type in a URL:")

URL = input()



#This strips out everthing leaving only the TLD (Future scope function)

def strip_domain(URL):

    global domain_name

    domain_name = get_fld(URL)

strip_domain(URL)





#This makes the request, and cleans up the source code

def connection(URL):

        r = requests.get(URL)

        status = r.status_code

        sourcecode = r.text

        soup = BeautifulSoup(sourcecode,features="html.parser")

        cleanupcode = soup.prettify()



        #This Strips the Anchor tags and adds them to the links array

        links = 

        for link in soup.findAll('a', attrs={'href': re.compile("^http://")}):

              links.append(link.get('href'))



        #This writes our clean anchor tags to a file

        with open('source.txt', 'w') as f:            

                for item in links:

                    f.write("%sn" % item)



connection(URL)

The exact code issue is around the "for link in soup.find" section.
I have been trying to parse the array for anchor tags the only contain the base domain, which is the global var "domain_name" so that it only writes the relevant links to the source txt file.

google.com accepted

google.com/file accepted

maps.google.com not written

If someone could assist me or point me in the right direction i'd appreciate it.
I was also thinking it would be possible to write every link to the source.txt file and then alter it after removing the 'out of scope' links, but really thought it more beneficial to do it without having to create additional code.

Additionally, i'm not the strongest with regex, but here is someone that my help.
This is some regex code to catch all variations of http, www, https

(^http://+|www.|https://)

To this I was going to append

.*{}'.format(domain_name)

I provide two different situationes. Because i donot agree that href value is xxx.com. Actually you will gain three or four or more kinds of href value, such as /file, folder/file, etc. So you have to transform relative path to absolute path, otherwise, you can not gather all of urls.

Regex: (/{2}([w]+.)?)([a-z.]+)(?=/?)

1. 
   (/{2}([w]+.)?) Matching non-main parts start from //


2. 
   ([a-z.]+)(?=/?) Match all specified character until we got /, we ought not to use .*(over-match)

My Code

import re



_input = "http://www.google.com/blabla"





all_part = re.findall(r"(/{2}([w]+.)?)([a-z.]+)(?=/?)",_input)[0]

_partA = all_part[2] # google.com

_partB = "".join(all_part[1:]) # www.google.com

print(_partA,_partB)



site = [

    "google.com",

    "google.com/file",

    "maps.google.com"

]

href = [

    "https://www.google.com",

    "https://www.google.com/file",

    "http://maps.google.com"

]

for ele in site:

    if re.findall("^{}/?".format(_partA),ele):

        print(ele)



for ele in href:

    if re.findall("{}/?".format(_partB),ele):

        print(ele)