16 Commits

Author SHA1 Message Date
KrakenTech 174071e472 Merge pull request #3 from Kraken-OffSec/hunter.io-subcommand
Hunter.io subcommand
2025-05-17 00:02:40 -04:00
Ar1ste1a 1f20cff41b Added hunter.io functions and file writes. 2025-05-17 00:01:48 -04:00
Ar1ste1a 2caccbee9d Added hunter.io functions and file writes. 2025-05-16 23:46:55 -04:00
Ar1ste1a 59ca1d4e92 Updated README.md
updated typos

added 'easy time' upgrade to log filtering
2025-05-16 20:54:16 -04:00
Ar1ste1a ad4c9197a2 Updated README.md
updated typos

added 'easy time' upgrade to log filtering
2025-05-16 20:53:53 -04:00
Ar1ste1a fccb213cf3 Updated README.md
updated typos

added 'easy time' upgrade to log filtering
2025-05-16 20:51:29 -04:00
Ar1ste1a 61e777e379 Updated README.md
updated typos

added 'easy time' upgrade to log filtering
2025-05-16 20:50:43 -04:00
Ar1ste1a 0b5a4bfea0 Updated README.md
updated typos

added 'easy time' upgrade to log filtering
2025-05-16 20:48:53 -04:00
Ar1ste1a 40e583b787 Updated README.md
updated typos

added 'easy time' upgrade to log filtering
2025-05-16 20:26:14 -04:00
KrakenTech 375aac0fca Merge pull request #2 from Kraken-OffSec/fixes-for-accidental-find-and-replace
Updated README.md
2025-05-16 20:22:27 -04:00
Ar1ste1a 32150ce6ee Updated README.md
updated typos

added 'easy time' upgrade to log filtering
2025-05-16 20:21:36 -04:00
Evan Hosinski 91fd75abe2 Updated table to include lookup 2025-05-16 18:46:16 -04:00
Evan Hosinski cc3016c5fb Updated table to include lookup 2025-05-16 18:45:24 -04:00
KrakenTech 86ceeba6d4 Merge pull request #1 from Kraken-OffSec/expand-database-to-include-whois
Added whois and lookups to queryable columnns and tables.
2025-05-16 16:30:07 -04:00
Evan Hosinski e8e8cede33 Added whois and lookups to queryable columnns and tables.
Removed unused db.go file.
2025-05-16 16:29:33 -04:00
KrakenTech f5a5f07997 Update Makefile 2025-05-16 15:57:38 -04:00
35 changed files with 2925 additions and 615 deletions
+1
View File
@@ -1,2 +1,3 @@
.idea/*
build/*
.DS_Store
Binary file not shown.

After

Width:  |  Height:  |  Size: 2.1 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.0 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.5 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 187 KiB

After

Width:  |  Height:  |  Size: 188 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 211 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 495 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 996 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.5 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 849 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 148 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.1 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 604 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 91 KiB

After

Width:  |  Height:  |  Size: 100 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 116 KiB

After

Width:  |  Height:  |  Size: 116 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 32 KiB

After

Width:  |  Height:  |  Size: 33 KiB

+1 -1
View File
@@ -16,7 +16,7 @@ PLATFORMS=linux darwin windows
ARCHS=amd64 arm64
# Version info from git tag or default
VERSION=$(shell git describe --tags 2>/dev/null || echo "v1.0.1")
VERSION=$(shell git describe --tags 2>/dev/null || echo "v1.2.1")
.PHONY: all clean build build-all
+123 -11
View File
@@ -76,8 +76,8 @@ To configure the database location:
### Simple Query
Dehasher can be used simply for example to query for credentials matching a given email domain.
``` go
# Provide credentials for emails matching @target.com
dehasher api -D @target.com -C
# Provide credentials for domains matching target.com
dehasher api -D target.com -C
```
### Simple Credentials Query
@@ -91,8 +91,8 @@ dehasher api -E @target.com -C
Dehasher is capable of handling multiple queries for the same field.
This is useful for when you want to search for multiple domains, or multiple usernames.
``` go
# Provide credentials for emails matching @target.com and @target2.com
dehasher api -E @target.com,@target2.com -C
# Provide credentials for domains matching target.com and target2.com, retrieving only credentials
dehasher api -D target.com,target2.com -C
```
### Wildcard Query
@@ -106,6 +106,20 @@ An asterisk can be used to denote multiple characters, and a question mark can b
dehasher api -E @target?.com -C -W
```
### Email Query
Dehashed has dictated that emails should be searched in the following format:
`email:target.name&domain:target.com`.
As such, to query an email, please use the following format (note, wildcard is not required but can be useful):
``` go
# Provide credentials for emails matching target.*@target.com
dehasher api -W -E 'target*' -D target.com
```
You may also query the domain and find emails as well
``` go
# Provide credentials for emails matching target.com
dehasher api -D target.com -C
```
### Regex Query
Dehasher is capable of handling regex queries.
@@ -113,7 +127,7 @@ Simply denote regex queries with the `-R` flag.
Place all regex queries in quotes with the corresponding query flag in single quotes.
``` go
# Return matches for emails matching this given regex query
dehasher api -R -e '[a-zA-Z0-9]+(?:\.[a-zA-Z0-9]+)?@target.com'
dehasher api -R -E '[a-zA-Z0-9]+(?:\.[a-zA-Z0-9]+)?@target.com'
```
### Output Text (default JSON)
@@ -126,7 +140,7 @@ Dehasher currently supports JSON, YAML, XML, and TEXT output formats.
dehasher api -U admin -o admins_file -f txt
```
<hr></hr>
---
## 🌐 WhoIs Lookups
Dehasher supports WHOIS lookups, history searches, reverse WHOIS searches, IP lookups, MX lookups, NS lookups, and subdomain scans.
@@ -196,16 +210,85 @@ dehasher whois -d google.com -s
---
## 🌐 Hunter.io
Dehasher supports Hunter.io lookups.
Hunter.io lookups require a separate API Key from the Dehashed API.
This can be set using the `set-hunter` command.
```bash
# Set the Hunter.io API key
dehasher set-hunter <redacted>
```
### Domain Search
Dehasher can perform a domain search for a given domain.
This provides a list of all emails that match the given query.
![Alt text](.img/hunter_domain_search.png "Hunter.io Domain Search")
```bash
# Perform a Hunter.io domain search for example.com
dehasher hunter -d example.com -D
```
### Email Finder
Dehasher can perform an email finder search for a given domain, first name, and last name.
This provides a list of all emails that match the given query.
![Alt text](.img/hunter_email_finder.png "Hunter.io Email Finder")
```bash
# Perform a Hunter.io email finder search for example.com
dehasher hunter -d example.com -F John -L Doe -E
```
### Email Verification
Dehasher can perform an email verification search for a given email.
This provides a list of all emails that match the given query.
![Alt text](.img/email_verification.png "Hunter.io Email Verification")
```bash
# Perform a Hunter.io email verification search for example@target.com
dehasher hunter -e example@target.com -V
```
### Company Enrichment
Dehasher can perform a company enrichment search for a given domain.
This provides a list of all emails that match the given query.
![Alt text](.img/company_enrichment.png "Hunter.io Company Enrichment")
```bash
# Perform a Hunter.io company enrichment search for example.com
dehasher hunter -d example.com -C
```
### Person Enrichment
Dehasher can perform a person enrichment search for a given email.
This provides a list of all emails that match the given query.
![Alt text](.img/person_enrichment.png "Hunter.io Person Enrichment")
```bash
# Perform a Hunter.io person enrichment search for example@target.com
dehasher hunter -e example@target.com -P
```
### Combined Enrichment
Dehasher can perform a combined enrichment search for a given email.
This provides a list of all emails that match the given query.
![Alt text](.img/combined_enrichment_1.png "Hunter.io Combined Enrichment")
![Alt text](.img/combined_enrichment_2.png "Hunter.io Combined Enrichment")
```bash
# Perform a Hunter.io combined enrichment search for example@target.com
dehasher hunter -e example@target.com -B
```
---
## 📊 Database Querying
Dehasher stores query results in a local database.
This database can be queried for previous results.
This is useful for when you want to query for specific information.
This database also includes WhoIs Information and Subdomain Scan results, but does **not** include historical lookups.
## Simple Query
#### It's possible to query the database using shorthand and without knowing any SQL at all.
#### The following queries the results table where username is not null, only showing the username, email and password columns.
![Alt text](.img/simple_query_db.png "Simple Query")
#### You may also add in a simple query using the `-q` flag. The following displays a 'LIKE' clause on the email column.
#### Note the '%\<clause\>%' is still required.
![Alt text](.img/simple_where.png "Simple Query")
Dehasher supports querying the database for previous results.
This is useful for when you want to query for specific information.
```bash
# Query the database for all results containing the word 'admin' in the username
dehasher query -t results -q "username LIKE '%admin%'"
@@ -238,11 +321,21 @@ dehasher query -a
The current tables available for query are:
- results
- Results from a dehashed query
- creds
- Credentials parsed from dehashed results
- whois
- Results from a whois record lookup
- subdomains
- history
- Subdomains discovered in a whois subdomain scan
- runs
- Previous query runs to the dehashed API
- lookup
- Results of any Whois NS, MX, or IP lookup
- hunter_domain
- Results from a hunter.io domain search
- hunter_email
- Results extracted from a domain saerch and email finder.
---
@@ -260,16 +353,35 @@ dehasher export -t results -q "username LIKE '%admin%'" -o admins_file -f txt
Dehasher uses the `zap` logging library for logging. The logs are stored in `~/.local/share/Dehasher/logs`.
The logs can be easily queried from the Dehasher CLI.
### Logs Dates
#### Dehasher utilized 'easy time' to determine the appropriate time for a given query.
![Alt text](.img/easy_time_parsing.png "Easy Time")
#### You may also used dates mixed with easy time to perform queries.
![Alt text](.img/mixed_time_query.png "Mixed Time")
#### The following formats are supported:
- `last 24 hours`
- `last 2 days`
- `30 minutes ago`
- `45 seconds ago`
- `1 week ago`
- `05-01-2025`
- `05/01/2025`
- `05/01/25`
- `05-01-25`
- `May 01, 2025`
```bash
# Show the last 10 logs
dehasher logs -l 10
# Show logs from the last 24 hours
dehasher logs -s "24 hours ago"
dehasher logs -s "last 24 hours"
# Show logs from the last 24 hours with a severity of error or fatal
dehasher logs -s "24 hours ago" -v error,fatal
dehasher logs -s "05-01-2025" -v error,fatal
```
## 🎉 Sample Run
```bash
ar1ste1a@kali:~$ dehasher api -D <redacted>.com -o <redacted> -f json
+19 -6
View File
@@ -2,14 +2,16 @@ package cmd
import (
"dehasher/internal/badger"
"dehasher/internal/debug"
"dehasher/internal/dehashed"
"dehasher/internal/sqlite"
"fmt"
"github.com/spf13/cobra"
"go.uber.org/zap"
)
func init() {
// Add query command to root command
// Add api command to root command
rootCmd.AddCommand(apiCmd)
// Add flags specific to api command
@@ -23,7 +25,7 @@ func init() {
apiCmd.Flags().StringVarP(&outputFormat, "format", "f", "json", "Output format (json, yaml, xml, txt)")
apiCmd.Flags().StringVarP(&outputFile, "output", "o", "query", "File to output results to including extension")
apiCmd.Flags().StringVarP(&usernameQuery, "username", "U", "", "Username query")
apiCmd.Flags().StringVarP(&emailQuery, "email-query", "E", "", "Email query")
apiCmd.Flags().StringVarP(&emailQuery, "email-query", "E", "", "HunterEmail query")
apiCmd.Flags().StringVarP(&ipQuery, "ip", "I", "", "IP address query")
apiCmd.Flags().StringVarP(&domainQuery, "domain", "D", "", "Domain query")
apiCmd.Flags().StringVarP(&passwordQuery, "password", "P", "", "Password query")
@@ -71,7 +73,7 @@ var (
Short: "Query the Dehashed API",
Long: `Query the Dehashed API for emails, usernames, passwords, hashes, IP addresses, and names.`,
Run: func(cmd *cobra.Command, args []string) {
key := getStoredApiKey()
key := getDehashedApiKey()
// Validate credentials
if key == "" {
@@ -116,12 +118,23 @@ var (
dehasher.Start()
fmt.Println("\n[*] Completing Process")
sqlite.StoreQueryOptions(queryOptions)
err := sqlite.StoreQueryOptions(queryOptions)
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to store query options")
debug.PrintError(err)
}
zap.L().Error("store_query_options",
zap.String("message", "failed to store query options"),
zap.Error(err),
)
fmt.Printf("Error storing query options: %v\n", err)
}
},
}
)
// Helper functions to get stored API credentials
func getStoredApiKey() string {
return badger.GetKey()
func getDehashedApiKey() string {
return badger.GetDehashedKey()
}
+429
View File
@@ -0,0 +1,429 @@
package cmd
import (
"dehasher/internal/badger"
"dehasher/internal/debug"
"dehasher/internal/export"
"dehasher/internal/files"
hunter "dehasher/internal/hunter.io"
"dehasher/internal/pretty"
"fmt"
"github.com/spf13/cobra"
"go.uber.org/zap"
"time"
)
func init() {
// Add hunter command to root command
rootCmd.AddCommand(hunterCmd)
// Add flags specific to hunter command
hunterCmd.Flags().StringVarP(&hunterDomain, "domain", "d", "", "Domain to query")
hunterCmd.Flags().StringVarP(&hunterEmail, "email", "e", "", "Email to query")
hunterCmd.Flags().StringVarP(&hunterFirstName, "first-name", "F", "", "First name to query")
hunterCmd.Flags().StringVarP(&hunterLastName, "last-name", "L", "", "Last name to query")
hunterCmd.Flags().BoolVarP(&hunterDomainSearch, "domain-search", "D", false, "Search for domain")
hunterCmd.Flags().BoolVarP(&hunterEmailFind, "email-find", "E", false, "Find emails for user using domain, first name, and last name")
hunterCmd.Flags().BoolVarP(&hunterEmailVerify, "email-verify", "V", false, "Verify email")
hunterCmd.Flags().BoolVarP(&hunterCompanyEnrichmentDomain, "company-enrichment", "C", false, "Company enrichment for domain")
hunterCmd.Flags().BoolVarP(&hunterPersonEnrichmentEmail, "person-enrichment", "P", false, "Person enrichment for email")
hunterCmd.Flags().BoolVarP(&hunterCombinedEnrichmentEmail, "combined-enrichment", "B", false, "Combined Company and Person enrichment for email")
hunterCmd.Flags().StringVarP(&hunterOutputFormat, "format", "f", "json", "Output format (json, yaml, xml, txt)")
hunterCmd.Flags().StringVarP(&hunterOutputFile, "output", "o", "hunter", "File to output results to including extension")
// Add mutually exclusive flags to hunter command
hunterCmd.MarkFlagsMutuallyExclusive("email-find")
}
var (
// Hunter Commands Flags
hunterDomain string
hunterEmail string
hunterFirstName string
hunterLastName string
hunterDomainSearch bool
hunterEmailFind bool
hunterEmailVerify bool
hunterCompanyEnrichmentDomain bool
hunterPersonEnrichmentEmail bool
hunterCombinedEnrichmentEmail bool
hunterOutputFormat string
hunterOutputFile string
hunterCmd = &cobra.Command{
Use: "hunter",
Short: "Hunter.io API interaction",
Long: `Interact with the Hunter.io API for email and domain information.`,
Run: func(cmd *cobra.Command, args []string) {
if debugGlobal {
debug.PrintInfo("debug mode enabled")
zap.L().Info("hunter_debug",
zap.String("message", "debug mode enabled"),
)
}
// Flag Checks
if !hunterFlagCheck() {
return
}
if hunterOutputFile == "" {
if debugGlobal {
debug.PrintInfo("output file not specified, using default")
}
hunterOutputFile = "hunter_" + time.Now().Format("05_04_05")
}
if hunterOutputFormat == "" {
if debugGlobal {
debug.PrintInfo("output format not specified, using default")
}
hunterOutputFormat = "json"
}
fType := files.GetFileType(hunterOutputFormat)
if fType == files.UNKNOWN {
fmt.Println("[!] Error: Invalid output format. Must be 'json', 'xml', 'yaml', or 'txt'.")
return
}
if debugGlobal {
debug.PrintInfo("using output format: " + hunterOutputFormat)
}
fmt.Println("[*] Hunter.io API interaction [Beta]")
h := hunter.NewHunterIO(getHunterApiKey(), debugGlobal)
if hunterDomainSearch {
fmt.Println("[*] Performing domain search search...")
result, err := h.DomainSearch(hunterDomain)
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to perform domain search")
debug.PrintError(err)
}
zap.L().Error("hunter_domain_search",
zap.String("message", "failed to perform domain search"),
zap.Error(err),
)
fmt.Printf("Error performing domain search: %v\n", err)
return
}
// Write Hunter.io Domain Search Result to file
fmt.Printf("[*] Writing Hunter.io Domain Search Result to file: %s%s\n", hunterOutputFile, fType.Extension())
err = export.WriteHunterDomainToFile(result, hunterOutputFile, fType)
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to write hunter domain search to file")
debug.PrintError(err)
}
zap.L().Error("write_hunter_domain_search",
zap.String("message", "failed to write hunter domain search to file"),
zap.Error(err),
)
fmt.Printf("Error writing Hunter.io Domain Search Result to file: %v\n", err)
}
// Pretty Print Hunter.io Domain Search Result
fmt.Println("Domain Search Result:")
pretty.HunterDomainTree(hunterDomain, result)
return
}
if hunterEmailFind {
fmt.Println("[*] Performing email find search...")
result, err := h.EmailFinder(hunterDomain, hunterFirstName, hunterLastName)
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to perform email find")
debug.PrintError(err)
}
zap.L().Error("hunter_email_find",
zap.String("message", "failed to perform email find"),
zap.Error(err),
)
fmt.Printf("Error performing email find: %v\n", err)
return
}
// Write Hunter.io Email Finder Result to file
fmt.Printf("[*] Writing Hunter.io Email Finder Result to file: %s%s\n", hunterOutputFile, fType.Extension())
err = export.WriteHunterEmailToFile(result, hunterOutputFile, fType)
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to write hunter email find to file")
debug.PrintError(err)
}
zap.L().Error("write_hunter_email_find",
zap.String("message", "failed to write hunter email find to file"),
zap.Error(err),
)
fmt.Printf("Error writing Hunter.io Email Finder Result to file: %v\n", err)
}
fmt.Println("Email Find Result:")
var (
headers = []string{"Email", "Score", "Domain", "Accept All", "Position", "Twitter", "Linkedin", "Phone Number", "Company", "Sources", "Verification"}
rows [][]string
)
rows = append(rows, []string{
result.Email,
fmt.Sprintf("%d", result.Score),
result.Domain,
fmt.Sprintf("%t", result.AcceptAll),
result.Position,
result.Twitter,
result.LinkedinURL,
result.PhoneNumber,
result.Company,
fmt.Sprintf("%v", result.Sources),
fmt.Sprintf("%v", result.Verification),
})
pretty.Table(headers, rows)
return
}
if hunterEmailVerify {
fmt.Println("[*] Performing email verification search...")
result, err := h.EmailVerification(hunterEmail)
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to perform email verification")
debug.PrintError(err)
}
zap.L().Error("hunter_email_verification",
zap.String("message", "failed to perform email verification"),
zap.Error(err),
)
fmt.Printf("Error performing email verification: %v\n", err)
return
}
// Write Hunter.io Email Verification Result to file
fmt.Printf("[*] Writing Hunter.io Email Verification Result to file: %s%s\n", hunterOutputFile, fType.Extension())
err = export.WriteHunterEmailVerifyToFile(result, hunterOutputFile, fType)
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to write hunter email verification to file")
debug.PrintError(err)
}
zap.L().Error("write_hunter_email_verification",
zap.String("message", "failed to write hunter email verification to file"),
zap.Error(err),
)
fmt.Printf("Error writing Hunter.io Email Verification Result to file: %v\n", err)
}
// Pretty Print Hunter.io Email Verification Result
var (
headers = []string{"Email", "Status", "Result", "Score", "Regexp", "Gibberish", "Disposable", "Webmail", "MX Records", "SMTP Server", "SMTP Check", "Accept All", "Block", "Sources"}
rows [][]string
)
rows = append(rows, []string{
result.Email,
result.Status,
result.Result,
fmt.Sprintf("%d", result.Score),
fmt.Sprintf("%t", result.Regexp),
fmt.Sprintf("%t", result.Gibberish),
fmt.Sprintf("%t", result.Disposable),
fmt.Sprintf("%t", result.Webmail),
fmt.Sprintf("%t", result.MXRecords),
fmt.Sprintf("%t", result.SMTPServer),
fmt.Sprintf("%t", result.SMTPCheck),
fmt.Sprintf("%t", result.AcceptAll),
fmt.Sprintf("%t", result.Block),
fmt.Sprintf("%v", result.Sources),
})
fmt.Println("Email Verification Result:")
pretty.Table(headers, rows)
return
}
if hunterCompanyEnrichmentDomain {
fmt.Println("[*] Performing company enrichment search...")
result, err := h.CompanyEnrichment(hunterDomain)
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to perform company enrichment")
debug.PrintError(err)
}
zap.L().Error("hunter_company_enrichment",
zap.String("message", "failed to perform company enrichment"),
zap.Error(err),
)
fmt.Printf("Error performing company enrichment: %v\n", err)
return
}
// Write to file
fmt.Printf("[*] Writing Hunter.io Company Enrichment Result to file: %s%s\n", hunterOutputFile, fType.Extension())
err = export.WriteHunterCompanyEnrichmentToFile(result, hunterOutputFile, fType)
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to write hunter company enrichment to file")
debug.PrintError(err)
}
zap.L().Error("write_hunter_company_enrichment",
zap.String("message", "failed to write hunter company enrichment to file"),
zap.Error(err),
)
fmt.Printf("Error writing Hunter.io Company Enrichment Result to file: %v\n", err)
}
// Pretty Print Hunter.io Company Enrichment Result
fmt.Println("Company Enrichment Result:")
pretty.HunterCompanyEnrichmentTree(hunterDomain, result)
return
}
if hunterPersonEnrichmentEmail {
fmt.Println("[*] Performing person enrichment search...")
result, err := h.PersonEnrichment(hunterEmail)
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to perform person enrichment")
debug.PrintError(err)
}
zap.L().Error("hunter_person_enrichment",
zap.String("message", "failed to perform person enrichment"),
zap.Error(err),
)
fmt.Printf("Error performing person enrichment: %v\n", err)
return
}
// Write to file
fmt.Printf("[*] Writing Hunter.io Person Enrichment Result to file: %s%s\n", hunterOutputFile, fType.Extension())
err = export.WriteHunterPersonEnrichmentToFile(result, hunterOutputFile, fType)
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to write hunter person enrichment to file")
debug.PrintError(err)
}
zap.L().Error("write_hunter_person_enrichment",
zap.String("message", "failed to write hunter person enrichment to file"),
zap.Error(err),
)
fmt.Printf("Error writing Hunter.io Person Enrichment Result to file: %v\n", err)
}
// Pretty Print Hunter.io Person Enrichment Result
fmt.Println("Person Enrichment Result:")
pretty.HunterPersonEnrichmentTree(hunterEmail, result)
return
}
if hunterCombinedEnrichmentEmail {
fmt.Println("[*] Performing combined enrichment search...")
result, err := h.CombinedEnrichment(hunterEmail)
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to perform combined enrichment")
debug.PrintError(err)
}
zap.L().Error("hunter_combined_enrichment",
zap.String("message", "failed to perform combined enrichment"),
zap.Error(err),
)
fmt.Printf("Error performing combined enrichment: %v\n", err)
return
}
// Write to file
fmt.Printf("[*] Writing Hunter.io Combined Enrichment Result to file: %s%s\n", hunterOutputFile, fType.Extension())
err = export.WriteHunterCombinedEnrichmentToFile(result, hunterOutputFile, fType)
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to write hunter combined enrichment to file")
debug.PrintError(err)
}
zap.L().Error("write_hunter_combined_enrichment",
zap.String("message", "failed to write hunter combined enrichment to file"),
zap.Error(err),
)
fmt.Printf("Error writing Hunter.io Combined Enrichment Result to file: %v\n", err)
}
fmt.Println("Combined Enrichment Result:")
pretty.HunterCombinedEnrichmentTree(hunterEmail, result)
return
}
},
}
)
func hunterFlagCheck() bool {
if debugGlobal {
debug.PrintInfo("checking flags")
}
var optionSet bool
if hunterDomainSearch {
if hunterDomain == "" {
fmt.Println("Domain is required for domain search")
return false
}
optionSet = true
}
if hunterEmailVerify {
if hunterEmail == "" {
fmt.Println("Email is required for email verification")
return false
}
optionSet = true
}
if hunterCompanyEnrichmentDomain {
if hunterDomain == "" {
fmt.Println("Domain is required for company enrichment")
return false
}
optionSet = true
}
if hunterPersonEnrichmentEmail {
if hunterEmail == "" {
fmt.Println("Email is required for person enrichment")
return false
}
optionSet = true
}
if hunterCombinedEnrichmentEmail {
if hunterEmail == "" {
fmt.Println("Email is required for combined enrichment")
return false
}
optionSet = true
}
if hunterEmailFind {
if hunterFirstName == "" || hunterLastName == "" {
fmt.Println("First name and last name are required for email find")
return false
}
if hunterDomain == "" {
fmt.Println("Domain is required for email find")
return false
}
optionSet = true
}
if !optionSet {
fmt.Println("[!] No options selected")
return false
}
return true
}
// Helper functions to get stored API credentials
func getHunterApiKey() string {
return badger.GetHunterKey()
}
+11 -32
View File
@@ -1,6 +1,7 @@
package cmd
import (
"dehasher/internal/easyTime"
"dehasher/internal/pretty"
"encoding/json"
"fmt"
@@ -73,6 +74,11 @@ var (
allLogs = append(allLogs, filepath.Join(logsPath, "info.log"), filepath.Join(logsPath, "error.log"))
}
var timeChunk easyTime.TimeChunk
if logStartDate != "" {
timeChunk = easyTime.NewTimeChunk(logStartDate, logEndDate, debugGlobal)
}
var parsedLogs []LogEntry
for _, logFile := range allLogs {
// Read the log file
@@ -97,7 +103,7 @@ var (
continue
}
// Also unmarshal to get additional fields
// Unmarshal to get additional fields
if err := json.Unmarshal([]byte(line), &rawEntry); err != nil {
fmt.Printf("Error parsing raw log entry: %v\n", err)
continue
@@ -106,10 +112,10 @@ var (
// Parse the timestamp
parsedTime, err := time.Parse("2006-01-02T15:04:05.999-0700", entry.Timestamp)
if err != nil {
// Try alternative formats
// Try RFC3339
parsedTime, err = time.Parse(time.RFC3339, entry.Timestamp)
if err != nil {
// Try another format
// Try RFC3339Nano
parsedTime, err = time.Parse(time.RFC3339Nano, entry.Timestamp)
if err != nil {
fmt.Printf("Error parsing timestamp '%s': %v\n", entry.Timestamp, err)
@@ -133,22 +139,8 @@ var (
(logFatal && strings.EqualFold(entry.Level, "FATAL")) {
// Filter by date range if specified
if logStartDate != "" {
startDate, err := time.Parse("2006-01-02", logStartDate)
if err != nil {
fmt.Printf("Error parsing start date: %v\n", err)
} else if entry.ParsedTime.Before(startDate) {
continue
}
}
if logEndDate != "" {
endDate, err := time.Parse("2006-01-02", logEndDate)
// Add one day to include the end date
endDate = endDate.Add(24 * time.Hour)
if err != nil {
fmt.Printf("Error parsing end date: %v\n", err)
} else if entry.ParsedTime.After(endDate) {
if timeChunk.IsSet() {
if entry.ParsedTime.Before(timeChunk.StartTime) || entry.ParsedTime.After(timeChunk.EndTime) {
continue
}
}
@@ -211,16 +203,3 @@ const (
FATAL
UNKNOWN Severity = -1
)
func getSeverity(logLevel string) Severity {
switch logLevel {
case "INFO":
return INFO
case "ERROR":
return ERROR
case "FATAL":
return FATAL
default:
return UNKNOWN
}
}
+31 -15
View File
@@ -12,31 +12,47 @@ import (
// Map of available tables and their columns
var availableTables = map[string][]string{
"results": {
"id", "created_at", "updated_at", "deleted_at", "dehashed_id", "email", "ip_address", "username",
"password", "hashed_password", "hash_type", "name", "vin", "license_plate", "url", "social",
"cryptocurrency_address", "address", "phone", "company", "database_name",
},
"creds": {
"id", "created_at", "updated_at", "deleted_at", "email", "username", "password",
},
"whois": {
"id", "created_at", "updated_at", "deleted_at", "contact_email", "created_date",
"domain_name", "domain_name_ext", "expires_date", "status", "whois_server",
},
"subdomains": {
"id", "created_at", "updated_at", "deleted_at", "domain", "first_seen", "last_seen",
},
"history": {
"id", "created_at", "updated_at", "deleted_at", "domain_name", "domain_type",
"registrar_name", "whois_server", "created_date_iso8601", "updated_date_iso8601", "expires_date_iso8601",
//"history": {
// "id", "created_at", "updated_at", "deleted_at", "domain_name", "domain_type",
// "registrar_name", "whois_server", "created_date_iso8601", "updated_date_iso8601", "expires_date_iso8601",
//},
"lookup": {
"id", "created_at", "updated_at", "deleted_at", "search_term", "type", "first_seen", "last_visit",
"name",
},
// Query Options
"runs": {
"id", "created_at", "updated_at", "deleted_at", "max_records", "max_requests", "starting_page",
"output_format", "output_file", "regex_match", "wildcard_match", "username_query", "email_query",
"ip_query", "pass_query", "hash_query", "name_query", "domain_query", "vin_query", "license_plate_query",
"address_query", "phone_query", "social_query", "crypto_address_query", "print_balance", "creds_only",
},
"results": {
"id", "created_at", "updated_at", "deleted_at", "dehashed_id", "email", "ip_address", "username",
"password", "hashed_password", "hash_type", "name", "vin", "license_plate", "url", "social",
"cryptocurrency_address", "address", "phone", "company", "database_name",
},
"subdomains": {
"id", "created_at", "updated_at", "deleted_at", "domain", "first_seen", "last_seen",
},
"whois": {
"id", "created_at", "updated_at", "deleted_at", "audit", "contact_email", "created_date", "created_date_normalized",
"domain_name", "domain_name_ext", "estimated_domain_age", "expires_date", "expires_date_normalized", "footer", "header",
"name_servers", "parse_code", "raw_text", "registrant", "registrar_iana_id", "registrar_name", "registry_data",
"status", "stripped_text", "updated_date", "updated_date_normalized",
},
"hunter_domain": {
"id", "created_at", "updated_at", "deleted_at", "domain", "disposable", "webmail", "accept_all", "pattern",
"organization", "description", "industry", "twitter", "facebook", "linkedin", "instagram", "youtube",
"technologies", "country", "state", "city", "postal_code", "street", "headcount", "company_type", "emails", "linked_domains",
},
"hunter_email": {
"id", "created_at", "updated_at", "deleted_at", "value", "type", "confidence", "sources", "first_name", "last_name",
"position", "position_raw", "seniority", "department", "linkedin", "twitter", "phone_number", "verification_date", "verification_status",
},
}
// Function to list available tables and their columns
+37 -11
View File
@@ -16,7 +16,7 @@ var (
// rootCmd is the base command for the CLI.
rootCmd = &cobra.Command{
Use: "dehasher",
Short: `Dehasher is a cli tool for querying query.`,
Short: `Dehasher is a cli tool for querying the dehashed api.`,
Long: fmt.Sprintf(
"%s\n%s",
`
@@ -42,7 +42,7 @@ var (
––•–√\/––√\/––•––––•–√\/––√\/––•––––•–√\/––√\/––•––√\/––•––––•–√\/––√\/––•––
`,
),
Version: "v1.0",
Version: "v1.2.1",
}
)
@@ -63,24 +63,41 @@ func init() {
rootCmd.CompletionOptions.HiddenDefaultCmd = true
// Add global flags
rootCmd.PersistentFlags().BoolVar(&debugGlobal, "debug", false, "Show debugGlobal information")
rootCmd.PersistentFlags().BoolVar(&debugGlobal, "debug", false, "Show debug information")
// Add subcommands
rootCmd.AddCommand(setKeyCmd)
rootCmd.AddCommand(setDehashedKeyCmd)
rootCmd.AddCommand(setHunterKeyCmd)
rootCmd.AddCommand(setLocalDb)
}
// Command to set API key
var setKeyCmd = &cobra.Command{
Use: "set-key [key]",
Short: "Set and store API key",
var setDehashedKeyCmd = &cobra.Command{
Use: "set-dehashed [key]",
Short: "Set and store Dehashed.com API key",
Args: cobra.ExactArgs(1),
Run: func(cmd *cobra.Command, args []string) {
key := args[0]
// Store key in badger DB
err := storeApiKey(key)
err := storeDehashedApiKey(key)
if err != nil {
fmt.Printf("Error storing API key: %v\n", err)
fmt.Printf("Error storing Dehashed API key: %v\n", err)
return
}
fmt.Println("API key stored successfully")
},
}
var setHunterKeyCmd = &cobra.Command{
Use: "set-hunter [key]",
Short: "Set and store Hunter.io API key",
Args: cobra.ExactArgs(1),
Run: func(cmd *cobra.Command, args []string) {
key := args[0]
// Store key in badger DB
err := storeHunterApiKey(key)
if err != nil {
fmt.Printf("Error storing Hunter API key: %v\n", err)
return
}
fmt.Println("API key stored successfully")
@@ -116,8 +133,17 @@ var setLocalDb = &cobra.Command{
}
// Helper functions to store API credentials
func storeApiKey(key string) error {
err := badger.StoreKey(key)
func storeDehashedApiKey(key string) error {
err := badger.StoreDehashedKey(key)
if err != nil {
fmt.Printf("Error storing API key: %v\n", err)
return err
}
return nil
}
func storeHunterApiKey(key string) error {
err := badger.StoreHunterKey(key)
if err != nil {
fmt.Printf("Error storing API key: %v\n", err)
return err
+36 -111
View File
@@ -10,6 +10,7 @@ import (
"fmt"
"github.com/spf13/cobra"
"go.uber.org/zap"
"os"
"strings"
"time"
)
@@ -54,7 +55,7 @@ var (
Short: "Dehashed WHOIS lookups and reverse WHOIS searches",
Long: `Perform WHOIS lookups, history searches, reverse WHOIS searches, IP lookups, MX lookups, NS lookups, and subdomain scans.`,
Run: func(cmd *cobra.Command, args []string) {
key := getStoredApiKey()
key := getDehashedApiKey()
// Validate credentials
if key == "" {
@@ -97,15 +98,9 @@ var (
// Show credits if requested
if whoisShowCredits {
fmt.Println("[*] Getting WHOIS balance...")
balance, err := w.Balance()
if err != nil {
zap.L().Error("get_whois_credits",
zap.String("message", "failed to get whois balance"),
zap.Error(err),
)
fmt.Printf("Error getting WHOIS balance: %v\n", err)
if whoisShowCredits {
checkBalance(w)
}
fmt.Printf("WHOIS Credits: %d\n", balance)
}
// Check if domain is provided for history and subdomain scan
@@ -115,15 +110,7 @@ var (
return
}
if whoisShowCredits {
balance, err := w.Balance()
if err != nil {
zap.L().Error("get_whois_credits",
zap.String("message", "failed to get whois balance"),
zap.Error(err),
)
fmt.Printf("Error getting WHOIS balance: %v\n", err)
}
fmt.Println("WHOIS Credits: ", balance)
checkBalance(w)
}
}
@@ -147,19 +134,7 @@ var (
}
if whoisShowCredits {
balance, err := w.Balance()
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to get whois balance")
debug.PrintError(err)
}
zap.L().Error("get_whois_credits",
zap.String("message", "failed to get whois balance"),
zap.Error(err),
)
fmt.Printf("Error getting WHOIS balance: %v\n", err)
}
fmt.Println("WHOIS Credits: ", balance)
checkBalance(w)
}
// Fix the output format to use proper formatting
@@ -197,6 +172,7 @@ var (
}
if whoisHistory {
filename := whoisOutputFile + "_history"
fmt.Println("[*] Performing WHOIS history search...")
// Perform history search
historyRecords, err := w.WhoisHistory(whoisDomain)
@@ -212,26 +188,14 @@ var (
fmt.Printf("[!] Error performing WHOIS history lookup: %v\n", err)
} else {
if whoisShowCredits {
balance, err := w.Balance()
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to get whois balance")
debug.PrintError(err)
}
zap.L().Error("get_whois_credits",
zap.String("message", "failed to get whois balance"),
zap.Error(err),
)
fmt.Printf("Error getting WHOIS balance: %v\n", err)
}
fmt.Println("WHOIS Credits: ", balance)
checkBalance(w)
}
// Write history records to file if any
if len(historyRecords) > 0 {
fmt.Println("[*] Records Found: %d\n", len(historyRecords))
fmt.Println("[*] WHOIS History being written to file: %s%s\n", whoisOutputFile, fType.Extension())
writeErr := export.WriteWhoIsHistoryToFile(historyRecords, whoisOutputFile, fType)
writeErr := export.WriteWhoIsHistoryToFile(historyRecords, filename, fType)
if writeErr != nil {
if debugGlobal {
debug.PrintInfo("failed to write whois history to file")
@@ -270,23 +234,12 @@ var (
// Perform subdomain scan
if whoisSubdomainScan {
filename := whoisOutputFile + "_subdomains"
fmt.Println("[*] Performing WHOIS subdomain scan...")
subdomains, err := w.WhoisSubdomainScan(whoisDomain)
if whoisShowCredits {
balance, err := w.Balance()
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to get whois balance")
debug.PrintError(err)
}
zap.L().Error("get_whois_credits",
zap.String("message", "failed to get whois balance"),
zap.Error(err),
)
fmt.Printf("Error getting WHOIS balance: %v\n", err)
}
fmt.Println("WHOIS Credits: ", balance)
checkBalance(w)
}
if err != nil {
@@ -317,7 +270,7 @@ var (
// Write the subdomains to file if any
if len(subdomains) > 0 {
fmt.Printf("[*] Writing subdomains to file: %s%s\n", whoisOutputFile, fType.Extension())
err = export.WriteSubdomainsToFile(subdomains, whoisOutputFile, fType)
err = export.WriteSubdomainsToFile(subdomains, filename, fType)
if err != nil {
zap.L().Error("write_whois_subdomain",
zap.String("message", "failed to write whois subdomain to file"),
@@ -371,19 +324,7 @@ var (
// Get credits
if whoisShowCredits {
balance, err := w.Balance()
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to get whois balance")
debug.PrintError(err)
}
zap.L().Error("get_whois_credits",
zap.String("message", "failed to get whois balance"),
zap.Error(err),
)
fmt.Printf("Error getting WHOIS balance: %v\n", err)
}
fmt.Println("WHOIS Credits: ", balance)
checkBalance(w)
}
if len(result) == 0 {
@@ -445,19 +386,7 @@ var (
// Get credits
if whoisShowCredits {
balance, err := w.Balance()
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to get whois balance")
debug.PrintError(err)
}
zap.L().Error("get_whois_credits",
zap.String("message", "failed to get whois balance"),
zap.Error(err),
)
fmt.Printf("Error getting WHOIS balance: %v\n", err)
}
fmt.Println("WHOIS Credits: ", balance)
checkBalance(w)
}
if len(result) == 0 {
@@ -519,19 +448,7 @@ var (
// Get credits
if whoisShowCredits {
balance, err := w.Balance()
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to get whois balance")
debug.PrintError(err)
}
zap.L().Error("get_whois_credits",
zap.String("message", "failed to get whois balance"),
zap.Error(err),
)
fmt.Printf("Error getting WHOIS balance: %v\n", err)
}
fmt.Println("WHOIS Credits: ", balance)
checkBalance(w)
}
if len(result) == 0 {
@@ -625,19 +542,7 @@ var (
fmt.Println(result)
if whoisShowCredits {
balance, err := w.Balance()
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to get whois balance")
debug.PrintError(err)
}
zap.L().Error("get_whois_credits",
zap.String("message", "failed to get whois balance"),
zap.Error(err),
)
fmt.Printf("Error getting WHOIS balance: %v\n", err)
}
fmt.Println("WHOIS Credits: ", balance)
checkBalance(w)
}
return
}
@@ -647,3 +552,23 @@ var (
},
}
)
func checkBalance(w *whois.DehashedWhoIs) {
balance, err := w.Balance()
if err != nil {
if debugGlobal {
debug.PrintInfo("failed to get whois balance")
debug.PrintError(err)
}
zap.L().Error("get_whois_credits",
zap.String("message", "failed to get whois balance"),
zap.Error(err),
)
fmt.Printf("Error getting WHOIS balance: %v\n", err)
}
fmt.Println("WHOIS Credits: ", balance)
if balance == 0 {
fmt.Println("[!] No WHOIS credits remaining.")
os.Exit(0)
}
}
+39 -3
View File
@@ -100,7 +100,7 @@ func Close() {
}
}
func GetKey() string {
func GetDehashedKey() string {
var apiKey string
err := db.View(func(txn *badger.Txn) error {
@@ -124,6 +124,29 @@ func GetKey() string {
return apiKey
}
func GetHunterKey() string {
var apiKey string
err := db.View(func(txn *badger.Txn) error {
item, err := txn.Get([]byte("cfg:hunter_api_key"))
if err != nil {
return err // could be ErrKeyNotFound
}
return item.Value(func(val []byte) error {
apiKey = string(val)
return nil
})
})
if err != nil {
zap.L().Error("get_hunter_api_key",
zap.String("message", "failed to get hunter_api_key"),
zap.Error(err),
)
}
return apiKey
}
func GetUseLocalDB() bool {
var useLocal bool
@@ -162,13 +185,26 @@ func GetUseLocalDB() bool {
return useLocal
}
func StoreKey(apiKey string) error {
func StoreDehashedKey(apiKey string) error {
err := db.Update(func(txn *badger.Txn) error {
return txn.Set([]byte("cfg:api_key"), []byte(apiKey))
})
if err != nil {
zap.L().Error("set_api_key",
zap.String("message", "failed to set api_key"),
zap.String("message", "failed to set dehashed api_key"),
zap.Error(err),
)
}
return err
}
func StoreHunterKey(apiKey string) error {
err := db.Update(func(txn *badger.Txn) error {
return txn.Set([]byte("cfg:hunter_api_key"), []byte(apiKey))
})
if err != nil {
zap.L().Error("set_api_key",
zap.String("message", "failed to set hunter api_key"),
zap.Error(err),
)
}
+266
View File
@@ -0,0 +1,266 @@
package easyTime
import (
"dehasher/internal/debug"
"fmt"
"go.uber.org/zap"
"os"
"strconv"
"strings"
"time"
)
type TimeChunk struct {
StartTime time.Time
EndTime time.Time
set bool
}
func (tc *TimeChunk) isValid() bool {
if !tc.StartTime.IsZero() && !tc.EndTime.IsZero() && tc.StartTime.Before(tc.EndTime) {
tc.set = true
return true
}
tc.set = false
return false
}
func (tc *TimeChunk) IsSet() bool {
return tc.set
}
func NewTimeChunk(start, end string, debugOn bool) TimeChunk {
if debugOn {
debug.PrintInfo("parsing time chunk")
debug.PrintInfo(fmt.Sprintf("Start: %s, End: %s", start, end))
zap.L().Info("parsing time chunk",
zap.String("start", start),
zap.String("end", end),
)
}
if end == "" {
if debugOn {
debug.PrintInfo("no end time provided, using now")
}
end = "now"
}
tc := TimeChunk{
StartTime: parseUserTime(start),
EndTime: parseUserTime(end),
}
if debugOn {
debug.PrintInfo("checking if time chunk is valid")
debug.PrintInfo(fmt.Sprintf("Start: %s, End: %s", tc.StartTime, tc.EndTime))
}
if !tc.isValid() {
fmt.Println("[!] Invalid time chunk")
zap.L().Fatal("invalid_time_chunk",
zap.String("message", "invalid time chunk"),
)
os.Exit(1)
}
return tc
}
func parseUserTime(args string) time.Time {
args = strings.TrimSpace(args)
if strings.EqualFold(args, "now") {
return time.Now()
}
// Check if time contains a space, if so, it's in 'last 24 hours' format
if strings.Contains(args, " ") && !containsMonth(strings.Split(args, " ")) {
splitArgs := strings.Split(args, " ")
if len(splitArgs) == 0 {
fmt.Println("[!] No time provided")
zap.L().Fatal("no_time_provided",
zap.String("message", "no time provided"),
)
os.Exit(1)
} else if len(splitArgs) < 3 {
fmt.Println("[!] Invalid time format")
zap.L().Fatal("invalid_time_format",
zap.String("message", "invalid time format"),
)
os.Exit(1)
}
// Handle 'last 24 hours' format
var (
tense string
amount int
duration time.Duration
)
for _, arg := range splitArgs {
if isPasteTense(arg) {
tense = arg
} else if isNumber(arg) {
amount, _ = strconv.Atoi(arg)
} else if isDuration(arg) {
duration = getDuration(arg)
}
}
if tense == "" {
fmt.Println("[!] Invalid time format: tense not found")
zap.L().Fatal("invalid_time_format",
zap.String("message", "invalid time format"),
)
os.Exit(1)
} else if amount == 0 {
fmt.Println("[!] Invalid time format: amount not found")
zap.L().Fatal("invalid_time_format",
zap.String("message", "invalid time format"),
)
os.Exit(1)
} else if duration == 0 {
fmt.Println("[!] Invalid time format: duration not found")
zap.L().Fatal("invalid_time_format",
zap.String("message", "invalid time format"),
)
os.Exit(1)
}
// Return the appropriate time
if tense == "last" {
return time.Now().Add(-time.Duration(amount) * duration)
} else if tense == "ago" {
return time.Now().Add(-time.Duration(amount) * duration)
}
}
// Handle possible formats 'May 01, 2025', '05-01-2025', '05/01/2025', '05/01/25', '05-01-25'
var (
t time.Time
err error
found bool
)
possible := []string{"01-02-2006", "01/02/2006", "01/02/06", "01-02-06", "Jan 02, 2006", "Jan 2, 2006"}
for _, format := range possible {
t, err = time.Parse(format, args)
if err == nil {
found = true
break
}
}
if !found {
fmt.Println("[!] Invalid time format")
zap.L().Fatal("invalid_time_format",
zap.String("message", "invalid time format"),
)
os.Exit(1)
}
// Convert UTC time to local time
local, err := time.LoadLocation("Local")
if err != nil {
fmt.Println("[!] Error loading local timezone")
zap.L().Error("load_timezone",
zap.String("message", "failed to load local timezone"),
zap.Error(err),
)
return t
}
// Convert the parsed time to local time
return time.Date(
t.Year(),
t.Month(),
t.Day(),
t.Hour(),
t.Minute(),
t.Second(),
t.Nanosecond(),
local,
)
}
func isPasteTense(value string) bool {
for _, v := range []string{"last", "ago"} {
if strings.EqualFold(value, v) {
return true
}
}
return false
}
func isDuration(value string) bool {
for _, v := range []string{"hour", "hours", "minute", "minutes", "second", "seconds", "day", "days", "week", "weeks", "month", "months", "year", "years"} {
if strings.EqualFold(value, v) {
return true
}
}
return false
}
func isNumber(value string) bool {
_, err := strconv.Atoi(value)
return err == nil
}
func getDuration(timeBlock string) time.Duration {
timeBlock = strings.TrimSpace(strings.ToLower(timeBlock))
switch timeBlock {
case "hour":
return time.Hour
case "hours":
return time.Hour
case "minute":
return time.Minute
case "minutes":
return time.Minute
case "second":
return time.Second
case "seconds":
return time.Second
case "day":
return 24 * time.Hour
case "days":
return 24 * time.Hour
case "week":
return 7 * 24 * time.Hour
case "weeks":
return 7 * 24 * time.Hour
case "month":
return 30 * 24 * time.Hour
case "months":
return 30 * 24 * time.Hour
case "year":
return 365 * 24 * time.Hour
case "years":
return 365 * 24 * time.Hour
default:
fmt.Printf("[!] Unknown duration: %s", timeBlock)
zap.L().Fatal("unknown_duration",
zap.String("message", "unknown duration"),
zap.String("duration", timeBlock),
)
os.Exit(1)
}
return 0
}
func containsMonth(arr []string) bool {
for _, v := range arr {
if isMonth(v) {
return true
}
}
return false
}
func isMonth(value string) bool {
for _, v := range []string{"jan", "feb", "mar", "apr", "may", "jun", "jul", "aug", "sep", "oct", "nov", "dec"} {
if strings.EqualFold(value, v) {
return true
}
}
return false
}
+161
View File
@@ -0,0 +1,161 @@
package export
import (
"dehasher/internal/files"
"dehasher/internal/sqlite"
"encoding/json"
"encoding/xml"
"fmt"
"gopkg.in/yaml.v3"
"os"
)
func WriteHunterDomainToFile(result sqlite.HunterDomainData, outputFile string, fileType files.FileType) error {
var data []byte
var err error
switch fileType {
case files.JSON:
data, err = json.MarshalIndent(result, "", " ")
case files.XML:
data, err = xml.MarshalIndent(result, "", " ")
case files.YAML:
data, err = yaml.Marshal(result)
case files.TEXT:
data = []byte(result.String())
default:
return err
}
if err != nil {
return err
}
filePath := fmt.Sprintf("%s.%s", outputFile, fileType.String())
return os.WriteFile(filePath, data, 0644)
}
func WriteHunterEmailToFile(result sqlite.HunterEmailFinderData, outputFile string, fileType files.FileType) error {
var data []byte
var err error
switch fileType {
case files.JSON:
data, err = json.MarshalIndent(result, "", " ")
case files.XML:
data, err = xml.MarshalIndent(result, "", " ")
case files.YAML:
data, err = yaml.Marshal(result)
case files.TEXT:
data = []byte(result.String())
default:
return err
}
if err != nil {
return err
}
filePath := fmt.Sprintf("%s.%s", outputFile, fileType.String())
return os.WriteFile(filePath, data, 0644)
}
func WriteHunterEmailVerifyToFile(result sqlite.HunterEmailVerifyData, outputFile string, fileType files.FileType) error {
var data []byte
var err error
switch fileType {
case files.JSON:
data, err = json.MarshalIndent(result, "", " ")
case files.XML:
data, err = xml.MarshalIndent(result, "", " ")
case files.YAML:
data, err = yaml.Marshal(result)
case files.TEXT:
data = []byte(result.String())
default:
return err
}
if err != nil {
return err
}
filePath := fmt.Sprintf("%s.%s", outputFile, fileType.String())
return os.WriteFile(filePath, data, 0644)
}
func WriteHunterCompanyEnrichmentToFile(result sqlite.CompanyData, outputFile string, fileType files.FileType) error {
var data []byte
var err error
switch fileType {
case files.JSON:
data, err = json.MarshalIndent(result, "", " ")
case files.XML:
data, err = xml.MarshalIndent(result, "", " ")
case files.YAML:
data, err = yaml.Marshal(result)
case files.TEXT:
data = []byte(result.String())
default:
return err
}
if err != nil {
return err
}
filePath := fmt.Sprintf("%s.%s", outputFile, fileType.String())
return os.WriteFile(filePath, data, 0644)
}
func WriteHunterPersonEnrichmentToFile(result sqlite.PersonData, outputFile string, fileType files.FileType) error {
var data []byte
var err error
switch fileType {
case files.JSON:
data, err = json.MarshalIndent(result, "", " ")
case files.XML:
data, err = xml.MarshalIndent(result, "", " ")
case files.YAML:
data, err = yaml.Marshal(result)
case files.TEXT:
data = []byte(result.String())
default:
return err
}
if err != nil {
return err
}
filePath := fmt.Sprintf("%s.%s", outputFile, fileType.String())
return os.WriteFile(filePath, data, 0644)
}
func WriteHunterCombinedEnrichmentToFile(result sqlite.CombinedData, outputFile string, fileType files.FileType) error {
var data []byte
var err error
switch fileType {
case files.JSON:
data, err = json.MarshalIndent(result, "", " ")
case files.XML:
data, err = xml.MarshalIndent(result, "", " ")
case files.YAML:
data, err = yaml.Marshal(result)
case files.TEXT:
data = []byte(result.String())
default:
return err
}
if err != nil {
return err
}
filePath := fmt.Sprintf("%s.%s", outputFile, fileType.String())
return os.WriteFile(filePath, data, 0644)
}
+696
View File
@@ -0,0 +1,696 @@
package hunter_io
import (
"dehasher/internal/debug"
"dehasher/internal/sqlite"
"encoding/json"
"fmt"
"go.uber.org/zap"
"io"
"net/http"
"strings"
)
const (
DOMAIN_SEARCH = "https://api.hunter.io/v2/domain-search?domain={{domain}}&api_key={{apikey}}"
EMAIL_FINDER = "https://api.hunter.io/v2/email-finder?domain={{domain}}&first_name={{first_name}}&last_name={{last_name}}&api_key={{apikey}}"
EMAIL_VERIFICATION = "https://api.hunter.io/v2/email-verifier?email={{email}}&api_key={{apikey}}"
COMPANY_ENRICHMENT = "https://api.hunter.io/v2/companies/find?domain={{domain}}&api_key={{apikey}}"
PERSON_ENRICHMENT = "https://api.hunter.io/v2/people/find?email={{email}}&api_key={{apikey}}"
COMBINED_ENRICHMENT = "https://api.hunter.io/v2/combined/find?email={{email}}&api_key={{apikey}}"
)
type HunterIO struct {
apiKey string
debug bool
}
func NewHunterIO(apiKey string, debugEnabled bool) *HunterIO {
return &HunterIO{apiKey: apiKey, debug: debugEnabled}
}
func (h *HunterIO) DomainSearch(domain string) (sqlite.HunterDomainData, error) {
var hunterDomainData sqlite.HunterDomainData
if h.debug {
debug.PrintInfo("performing domain search")
zap.L().Info("hunter_domain_search_debug",
zap.String("message", "performing domain search"),
)
}
url := DOMAIN_SEARCH
url = strings.Replace(url, "{{domain}}", domain, -1)
url = strings.Replace(url, "{{apikey}}", h.apiKey, -1)
if h.debug {
debug.PrintInfo("performing request")
debug.PrintInfo(fmt.Sprintf("URL: %s\n", url))
zap.L().Info("hunter_domain_search_debug",
zap.String("message", "performing request"),
zap.String("url", url),
)
}
resp, err := http.Get(url)
if err != nil {
if h.debug {
debug.PrintInfo("failed to perform request")
debug.PrintError(err)
}
zap.L().Error("hunter_domain_search",
zap.String("message", "failed to perform request"),
zap.Error(err),
)
return hunterDomainData, err
}
defer resp.Body.Close()
b, err := io.ReadAll(resp.Body)
if err != nil {
if h.debug {
debug.PrintInfo("failed to read response body")
debug.PrintError(err)
}
zap.L().Error("hunter_domain_search",
zap.String("message", "failed to read response body"),
zap.Error(err),
)
return hunterDomainData, err
}
if resp.StatusCode != 200 {
if h.debug {
debug.PrintInfo("received error status code")
debug.PrintJson(fmt.Sprintf("Status Code: %d\n", resp.StatusCode))
debug.PrintJson(fmt.Sprintf("Body: %s\n", string(b)))
zap.L().Info("hunter_domain_search_debug",
zap.String("message", "received error status code"),
zap.Int("status_code", resp.StatusCode),
zap.String("body_error", string(b)),
)
}
zap.L().Error("hunter_domain_search",
zap.String("message", "received error status code"),
zap.Int("status_code", resp.StatusCode),
zap.String("body_error", string(b)),
)
return hunterDomainData, fmt.Errorf("received error status code: %d", resp.StatusCode)
}
if h.debug {
debug.PrintInfo("unmarshalled response body")
debug.PrintJson(fmt.Sprintf("Body: %s\n", string(b)))
zap.L().Info("hunter_domain_search_debug",
zap.String("message", "unmarshalled response body"),
zap.String("body", string(b)),
)
}
var hunterDomainSearchResult sqlite.HunterDomainSearchResult
err = json.Unmarshal(b, &hunterDomainSearchResult)
if err != nil {
if h.debug {
debug.PrintInfo("failed to unmarshal response body")
debug.PrintError(err)
}
zap.L().Error("hunter_domain_search",
zap.String("message", "failed to unmarshal response body"),
zap.Error(err),
)
return hunterDomainData, err
}
hunterDomainData = hunterDomainSearchResult.Data
// Create a list of email object associated with the domain
var emails []sqlite.HunterEmail
for _, email := range hunterDomainData.Emails {
emails = append(emails, sqlite.HunterEmail{
Domain: domain,
Value: email.Value,
Type: email.Type,
Confidence: email.Confidence,
Sources: email.Sources,
FirstName: email.FirstName,
LastName: email.LastName,
Position: email.Position,
PositionRaw: email.PositionRaw,
Seniority: email.Seniority,
Department: email.Department,
Linkedin: email.Linkedin,
Twitter: email.Twitter,
PhoneNumber: email.PhoneNumber,
Verification: email.Verification,
})
}
err = sqlite.StoreHunterEmails(emails)
if err != nil {
if h.debug {
debug.PrintInfo("failed to store hunter emails")
debug.PrintError(err)
}
zap.L().Error("store_hunter_emails",
zap.String("message", "failed to store hunter emails"),
zap.Error(err),
)
return hunterDomainData, err
}
err = sqlite.StoreHunterDomain(hunterDomainData)
if err != nil {
if h.debug {
debug.PrintInfo("failed to store hunter domain")
debug.PrintError(err)
}
zap.L().Error("store_hunter_domain",
zap.String("message", "failed to store hunter domain"),
zap.Error(err),
)
return hunterDomainData, err
}
return hunterDomainData, nil
}
func (h *HunterIO) EmailFinder(domain, firstName, lastName string) (sqlite.HunterEmailFinderData, error) {
var hunterEmailFinderData sqlite.HunterEmailFinderData
if h.debug {
debug.PrintInfo("performing email find")
zap.L().Info("hunter_email_find_debug",
zap.String("message", "performing email find"),
)
}
url := EMAIL_FINDER
url = strings.Replace(url, "{{domain}}", domain, -1)
url = strings.Replace(url, "{{first_name}}", firstName, -1)
url = strings.Replace(url, "{{last_name}}", lastName, -1)
url = strings.Replace(url, "{{apikey}}", h.apiKey, -1)
if h.debug {
debug.PrintInfo("performing request")
debug.PrintInfo(fmt.Sprintf("URL: %s\n", url))
zap.L().Info("hunter_email_find_debug",
zap.String("message", "performing request"),
zap.String("url", url),
)
}
resp, err := http.Get(url)
if err != nil {
if h.debug {
debug.PrintInfo("failed to perform request")
debug.PrintError(err)
}
zap.L().Error("hunter_email_find",
zap.String("message", "failed to perform request"),
zap.Error(err),
)
return hunterEmailFinderData, err
}
defer resp.Body.Close()
b, err := io.ReadAll(resp.Body)
if err != nil {
if h.debug {
debug.PrintInfo("failed to read response body")
debug.PrintError(err)
}
zap.L().Error("hunter_email_find",
zap.String("message", "failed to read response body"),
zap.Error(err),
)
return hunterEmailFinderData, err
}
if resp.StatusCode != 200 {
if h.debug {
debug.PrintInfo("received error status code")
debug.PrintJson(fmt.Sprintf("Status Code: %d\n", resp.StatusCode))
debug.PrintJson(fmt.Sprintf("Body: %s\n", string(b)))
zap.L().Info("hunter_email_find_debug",
zap.String("message", "received error status code"),
zap.Int("status_code", resp.StatusCode),
zap.String("body_error", string(b)),
)
}
zap.L().Error("hunter_email_find",
zap.String("message", "received error status code"),
zap.Int("status_code", resp.StatusCode),
zap.String("body_error", string(b)),
)
return hunterEmailFinderData, fmt.Errorf("received error status code: %d", resp.StatusCode)
}
if h.debug {
debug.PrintInfo("unmarshalled response body")
debug.PrintJson(fmt.Sprintf("Body: %s\n", string(b)))
zap.L().Info("hunter_email_find_debug",
zap.String("message", "unmarshalled response body"),
zap.String("body", string(b)),
)
}
var hunterEmailFinderResult sqlite.HunterEmailFinderResponse
err = json.Unmarshal(b, &hunterEmailFinderResult)
if err != nil {
if h.debug {
debug.PrintInfo("failed to unmarshal response body")
debug.PrintError(err)
}
zap.L().Error("hunter_email_find",
zap.String("message", "failed to unmarshal response body"),
zap.Error(err),
)
return hunterEmailFinderData, err
}
hunterEmailFinderData = hunterEmailFinderResult.Data
var hunterEmails []sqlite.HunterEmail
hunterEmails = append(hunterEmails, sqlite.HunterEmail{
Domain: hunterEmailFinderData.Domain,
Value: hunterEmailFinderData.Email,
Type: "personal",
Confidence: 100,
Sources: hunterEmailFinderData.Sources,
FirstName: hunterEmailFinderData.FirstName,
LastName: hunterEmailFinderData.LastName,
Position: hunterEmailFinderData.Position,
PositionRaw: "",
Seniority: "",
Department: "",
Linkedin: hunterEmailFinderData.LinkedinURL,
Twitter: hunterEmailFinderData.Twitter,
PhoneNumber: hunterEmailFinderData.PhoneNumber,
Verification: hunterEmailFinderData.Verification,
})
err = sqlite.StoreHunterEmails(hunterEmails)
if err != nil {
if h.debug {
debug.PrintInfo("failed to store hunter email finder")
debug.PrintError(err)
}
zap.L().Error("store_hunter_email_finder",
zap.String("message", "failed to store hunter email finder"),
zap.Error(err),
)
return hunterEmailFinderData, err
}
return hunterEmailFinderData, nil
}
func (h *HunterIO) EmailVerification(email string) (sqlite.HunterEmailVerifyData, error) {
var hunterEmailVerifyData sqlite.HunterEmailVerifyData
if h.debug {
debug.PrintInfo("performing email verification")
zap.L().Info("hunter_email_verification_debug",
zap.String("message", "performing email verification"),
)
}
url := EMAIL_VERIFICATION
url = strings.Replace(url, "{{email}}", email, -1)
url = strings.Replace(url, "{{apikey}}", h.apiKey, -1)
if h.debug {
debug.PrintInfo("performing request")
debug.PrintInfo(fmt.Sprintf("URL: %s\n", url))
zap.L().Info("hunter_email_verification_debug",
zap.String("message", "performing request"),
zap.String("url", url),
)
}
resp, err := http.Get(url)
if err != nil {
if h.debug {
debug.PrintInfo("failed to perform request")
debug.PrintError(err)
}
zap.L().Error("hunter_email_verification",
zap.String("message", "failed to perform request"),
zap.Error(err),
)
return hunterEmailVerifyData, err
}
defer resp.Body.Close()
b, err := io.ReadAll(resp.Body)
if err != nil {
if h.debug {
debug.PrintInfo("failed to read response body")
debug.PrintError(err)
}
zap.L().Error("hunter_email_verification",
zap.String("message", "failed to read response body"),
zap.Error(err),
)
return hunterEmailVerifyData, err
}
if resp.StatusCode != 200 {
if h.debug {
debug.PrintInfo("received error status code")
debug.PrintJson(fmt.Sprintf("Status Code: %d\n", resp.StatusCode))
debug.PrintJson(fmt.Sprintf("Body: %s\n", string(b)))
zap.L().Info("hunter_email_verification_debug",
zap.String("message", "received error status code"),
zap.Int("status_code", resp.StatusCode),
zap.String("body_error", string(b)),
)
}
zap.L().Error("hunter_email_verification",
zap.String("message", "received error status code"),
zap.Int("status_code", resp.StatusCode),
zap.String("body_error", string(b)),
)
return hunterEmailVerifyData, fmt.Errorf("received error status code: %d", resp.StatusCode)
}
if h.debug {
debug.PrintInfo("unmarshalled response body")
debug.PrintJson(fmt.Sprintf("Body: %s\n", string(b)))
zap.L().Info("hunter_email_verification_debug",
zap.String("message", "unmarshalled response body"),
zap.String("body", string(b)),
)
}
var hunterEmailVerifyResult sqlite.HunterEmailVerifyResponse
err = json.Unmarshal(b, &hunterEmailVerifyResult)
if err != nil {
if h.debug {
debug.PrintInfo("failed to unmarshal response body")
debug.PrintError(err)
}
zap.L().Error("hunter_email_verification",
zap.String("message", "failed to unmarshal response body"),
zap.Error(err),
)
return hunterEmailVerifyData, err
}
hunterEmailVerifyData = hunterEmailVerifyResult.Data
return hunterEmailVerifyData, nil
}
func (h *HunterIO) CompanyEnrichment(domain string) (sqlite.CompanyData, error) {
var companyData sqlite.CompanyData
if h.debug {
debug.PrintInfo("performing company enrichment")
zap.L().Info("hunter_company_enrichment_debug",
zap.String("message", "performing company enrichment"),
)
}
url := COMPANY_ENRICHMENT
url = strings.Replace(url, "{{domain}}", domain, -1)
url = strings.Replace(url, "{{apikey}}", h.apiKey, -1)
if h.debug {
debug.PrintInfo("performing request")
debug.PrintInfo(fmt.Sprintf("URL: %s\n", url))
zap.L().Info("hunter_company_enrichment_debug",
zap.String("message", "performing request"),
zap.String("url", url),
)
}
resp, err := http.Get(url)
if err != nil {
if h.debug {
debug.PrintInfo("failed to perform request")
debug.PrintError(err)
}
zap.L().Error("hunter_company_enrichment",
zap.String("message", "failed to perform request"),
zap.Error(err),
)
return companyData, err
}
defer resp.Body.Close()
b, err := io.ReadAll(resp.Body)
if err != nil {
if h.debug {
debug.PrintInfo("failed to read response body")
debug.PrintError(err)
}
zap.L().Error("hunter_company_enrichment",
zap.String("message", "failed to read response body"),
zap.Error(err),
)
return companyData, err
}
if resp.StatusCode != 200 {
if h.debug {
debug.PrintInfo("received error status code")
debug.PrintJson(fmt.Sprintf("Status Code: %d\n", resp.StatusCode))
debug.PrintJson(fmt.Sprintf("Body: %s\n", string(b)))
zap.L().Info("hunter_company_enrichment_debug",
zap.String("message", "received error status code"),
zap.Int("status_code", resp.StatusCode),
zap.String("body_error", string(b)),
)
}
zap.L().Error("hunter_company_enrichment",
zap.String("message", "received error status code"),
zap.Int("status_code", resp.StatusCode),
zap.String("body_error", string(b)),
)
return companyData, fmt.Errorf("received error status code: %d", resp.StatusCode)
}
if h.debug {
debug.PrintInfo("unmarshalled response body")
debug.PrintJson(fmt.Sprintf("Body: %s\n", string(b)))
zap.L().Info("hunter_company_enrichment_debug",
zap.String("message", "unmarshalled response body"),
zap.String("body", string(b)),
)
}
var hunterCompanyEnrichmentResult sqlite.HunterCompanyEnrichmentResponse
err = json.Unmarshal(b, &hunterCompanyEnrichmentResult)
if err != nil {
if h.debug {
debug.PrintInfo("failed to unmarshal response body")
debug.PrintError(err)
}
zap.L().Error("hunter_company_enrichment",
zap.String("message", "failed to unmarshal response body"),
zap.Error(err),
)
return companyData, err
}
companyData = hunterCompanyEnrichmentResult.Data
return companyData, nil
}
func (h *HunterIO) PersonEnrichment(email string) (sqlite.PersonData, error) {
var personData sqlite.PersonData
if h.debug {
debug.PrintInfo("performing person enrichment")
zap.L().Info("hunter_person_enrichment_debug",
zap.String("message", "performing person enrichment"),
)
}
url := PERSON_ENRICHMENT
url = strings.Replace(url, "{{email}}", email, -1)
url = strings.Replace(url, "{{apikey}}", h.apiKey, -1)
if h.debug {
debug.PrintInfo("performing request")
debug.PrintInfo(fmt.Sprintf("URL: %s\n", url))
zap.L().Info("hunter_person_enrichment_debug",
zap.String("message", "performing request"),
zap.String("url", url),
)
}
resp, err := http.Get(url)
if err != nil {
if h.debug {
debug.PrintInfo("failed to perform request")
debug.PrintError(err)
}
zap.L().Error("hunter_person_enrichment",
zap.String("message", "failed to perform request"),
zap.Error(err),
)
return personData, err
}
defer resp.Body.Close()
b, err := io.ReadAll(resp.Body)
if err != nil {
if h.debug {
debug.PrintInfo("failed to read response body")
debug.PrintError(err)
}
zap.L().Error("hunter_person_enrichment",
zap.String("message", "failed to read response body"),
zap.Error(err),
)
return personData, err
}
if resp.StatusCode != 200 {
if h.debug {
debug.PrintInfo("received error status code")
debug.PrintJson(fmt.Sprintf("Status Code: %d\n", resp.StatusCode))
debug.PrintJson(fmt.Sprintf("Body: %s\n", string(b)))
zap.L().Info("hunter_person_enrichment_debug",
zap.String("message", "received error status code"),
zap.Int("status_code", resp.StatusCode),
zap.String("body_error", string(b)),
)
}
zap.L().Error("hunter_person_enrichment",
zap.String("message", "received error status code"),
zap.Int("status_code", resp.StatusCode),
zap.String("body_error", string(b)),
)
return personData, fmt.Errorf("received error status code: %d", resp.StatusCode)
}
if h.debug {
debug.PrintInfo("unmarshalled response body")
debug.PrintJson(fmt.Sprintf("Body: %s\n", string(b)))
zap.L().Info("hunter_person_enrichment_debug",
zap.String("message", "unmarshalled response body"),
zap.String("body", string(b)),
)
}
var hunterPersonEnrichmentResult sqlite.HunterPersonEnrichmentResponse
err = json.Unmarshal(b, &hunterPersonEnrichmentResult)
if err != nil {
if h.debug {
debug.PrintInfo("failed to unmarshal response body")
debug.PrintError(err)
}
zap.L().Error("hunter_person_enrichment",
zap.String("message", "failed to unmarshal response body"),
zap.Error(err),
)
return personData, err
}
personData = hunterPersonEnrichmentResult.Data
return personData, nil
}
func (h *HunterIO) CombinedEnrichment(email string) (sqlite.CombinedData, error) {
var combinedData sqlite.CombinedData
if h.debug {
debug.PrintInfo("performing combined enrichment")
zap.L().Info("hunter_combined_enrichment_debug",
zap.String("message", "performing combined enrichment"),
)
}
url := COMBINED_ENRICHMENT
url = strings.Replace(url, "{{email}}", email, -1)
url = strings.Replace(url, "{{apikey}}", h.apiKey, -1)
if h.debug {
debug.PrintInfo("performing request")
debug.PrintInfo(fmt.Sprintf("URL: %s\n", url))
zap.L().Info("hunter_combined_enrichment_debug",
zap.String("message", "performing request"),
zap.String("url", url),
)
}
resp, err := http.Get(url)
if err != nil {
if h.debug {
debug.PrintInfo("failed to perform request")
debug.PrintError(err)
}
zap.L().Error("hunter_combined_enrichment",
zap.String("message", "failed to perform request"),
zap.Error(err),
)
return combinedData, err
}
defer resp.Body.Close()
b, err := io.ReadAll(resp.Body)
if err != nil {
if h.debug {
debug.PrintInfo("failed to read response body")
debug.PrintError(err)
}
zap.L().Error("hunter_combined_enrichment",
zap.String("message", "failed to read response body"),
zap.Error(err),
)
return combinedData, err
}
if resp.StatusCode != 200 {
if h.debug {
debug.PrintInfo("received error status code")
debug.PrintJson(fmt.Sprintf("Status Code: %d\n", resp.StatusCode))
debug.PrintJson(fmt.Sprintf("Body: %s\n", string(b)))
zap.L().Info("hunter_combined_enrichment_debug",
zap.String("message", "received error status code"),
zap.Int("status_code", resp.StatusCode),
zap.String("body_error", string(b)),
)
}
zap.L().Error("hunter_combined_enrichment",
zap.String("message", "received error status code"),
zap.Int("status_code", resp.StatusCode),
zap.String("body_error", string(b)),
)
return combinedData, fmt.Errorf("received error status code: %d", resp.StatusCode)
}
if h.debug {
debug.PrintInfo("unmarshalled response body")
debug.PrintJson(fmt.Sprintf("Body: %s\n", string(b)))
zap.L().Info("hunter_combined_enrichment_debug",
zap.String("message", "unmarshalled response body"),
zap.String("body", string(b)),
)
}
var hunterCombinedEnrichmentResult sqlite.HunterCombinedEnrichmentResponse
err = json.Unmarshal(b, &hunterCombinedEnrichmentResult)
if err != nil {
if h.debug {
debug.PrintInfo("failed to unmarshal response body")
debug.PrintError(err)
}
zap.L().Error("hunter_combined_enrichment",
zap.String("message", "failed to unmarshal response body"),
zap.Error(err),
)
return combinedData, err
}
combinedData = hunterCombinedEnrichmentResult.Data
return combinedData, nil
}
+242 -1
View File
@@ -70,7 +70,7 @@ func WhoIsTree(root string, record sqlite.WhoisRecord) {
technicalContactTree.Child("Telephone: " + record.TechnicalContact.Telephone)
// Root Tree Children
rootTree.Child("Contact Email: " + record.ContactEmail)
rootTree.Child("Contact HunterEmail: " + record.ContactEmail)
rootTree.Child("Created Date: " + record.CreatedDate)
rootTree.Child("Created Date Normalized: " + record.CreatedDateNormalized)
rootTree.Child("Domain Name: " + record.DomainName)
@@ -102,3 +102,244 @@ func WhoIsTree(root string, record sqlite.WhoisRecord) {
// Print Tree
fmt.Println(rootTree)
}
func HunterDomainTree(root string, record sqlite.HunterDomainData) {
enumeratorStyle := lipgloss.NewStyle().Foreground(purple).MarginRight(1)
rootStyle := lipgloss.NewStyle().Foreground(lipgloss.Color("63"))
itemStyle := lipgloss.NewStyle().Foreground(gray)
rootTree := tree.Root(root)
// Root Tree Children
rootTree.Child("Domain: " + record.Domain)
rootTree.Child("Disposable: " + fmt.Sprintf("%t", record.Disposable))
rootTree.Child("Webmail: " + fmt.Sprintf("%t", record.Webmail))
rootTree.Child("Accept All: " + fmt.Sprintf("%t", record.AcceptAll))
rootTree.Child("Pattern: " + record.Pattern)
rootTree.Child("Organization: " + record.Organization)
rootTree.Child("Description: " + record.Description)
rootTree.Child("Industry: " + record.Industry)
rootTree.Child("Twitter: " + record.Twitter)
rootTree.Child("Facebook: " + record.Facebook)
rootTree.Child("Linkedin: " + record.Linkedin)
rootTree.Child("Instagram: " + record.Instagram)
rootTree.Child("Youtube: " + record.Youtube)
techTree := tree.Root("Technologies")
for _, tech := range record.Technologies {
techTree.Child(tech)
}
rootTree.Child(techTree)
rootTree.Child("Country: " + record.Country)
rootTree.Child("State: " + record.State)
rootTree.Child("City: " + record.City)
rootTree.Child("Postal Code: " + record.PostalCode)
rootTree.Child("Street: " + record.Street)
rootTree.Child("Headcount: " + record.Headcount)
rootTree.Child("Company Type: " + record.CompanyType)
emailTree := tree.Root("Emails")
for _, email := range record.Emails {
emailTree.Child(email.ToTree())
}
rootTree.Child(emailTree)
linkedDomainTree := tree.Root("Linked Domains")
for _, domain := range record.LinkedDomains {
linkedDomainTree.Child(domain)
}
rootTree.Child(linkedDomainTree)
// Styles
rootTree.Enumerator(tree.RoundedEnumerator)
rootTree.EnumeratorStyle(enumeratorStyle)
rootTree.RootStyle(rootStyle)
rootTree.ItemStyle(itemStyle)
// Print Tree
fmt.Println(rootTree)
}
func HunterCompanyEnrichmentTree(root string, record sqlite.CompanyData) {
enumeratorStyle := lipgloss.NewStyle().Foreground(purple).MarginRight(1)
rootStyle := lipgloss.NewStyle().Foreground(lipgloss.Color("63"))
itemStyle := lipgloss.NewStyle().Foreground(gray)
rootTree := tree.Root(root)
// Root Tree Children
rootTree.Child("ID: " + record.ID)
rootTree.Child("Name: " + record.Name)
rootTree.Child("Legal Name: " + record.LegalName)
rootTree.Child("Domain: " + record.Domain)
rootTree.Child(record.DomainAliasesTree())
rootTree.Child(record.SiteTree())
rootTree.Child(record.CategoryTree())
rootTree.Child(record.TagsTree())
rootTree.Child("Description: " + record.Description)
rootTree.Child("Founded Year: " + fmt.Sprintf("%d", record.FoundedYear))
rootTree.Child("Location: " + record.Location)
rootTree.Child("Time Zone: " + record.TimeZone)
rootTree.Child("UTC Offset: " + fmt.Sprintf("%d", record.UTCOffset))
rootTree.Child(record.GeoTree())
rootTree.Child("Logo: " + record.Logo)
rootTree.Child(record.FacebookTree())
rootTree.Child(record.LinkedInTree())
rootTree.Child(record.TwitterTree())
rootTree.Child(record.CrunchbaseTree())
rootTree.Child(record.YouTubeTree())
rootTree.Child("Email Provider: " + record.EmailProvider)
rootTree.Child("Type: " + record.Type)
rootTree.Child("Ticker: " + record.Ticker)
rootTree.Child(record.IdentifiersTree())
rootTree.Child("Phone: " + record.Phone)
rootTree.Child(record.MetricsTree())
rootTree.Child("Indexed At: " + record.IndexedAt)
rootTree.Child(record.TechTree())
rootTree.Child(record.TechCategoriesTree())
rootTree.Child(record.ParentTree())
rootTree.Child(record.UltimateParentTree())
// Styles
rootTree.Enumerator(tree.RoundedEnumerator)
rootTree.EnumeratorStyle(enumeratorStyle)
rootTree.RootStyle(rootStyle)
rootTree.ItemStyle(itemStyle)
// Print Tree
fmt.Println(rootTree)
}
func HunterPersonEnrichmentTree(root string, record sqlite.PersonData) {
enumeratorStyle := lipgloss.NewStyle().Foreground(purple).MarginRight(1)
rootStyle := lipgloss.NewStyle().Foreground(lipgloss.Color("63"))
itemStyle := lipgloss.NewStyle().Foreground(gray)
rootTree := tree.Root(root)
// Root Tree Children
rootTree.Child("ID: " + record.ID)
rootTree.Child(record.NameTree())
rootTree.Child("Email: " + record.Email)
rootTree.Child("Location: " + record.Location)
rootTree.Child("Time Zone: " + record.TimeZone)
rootTree.Child("UTC Offset: " + fmt.Sprintf("%d", record.UTCOffset))
rootTree.Child(record.GeoTree())
rootTree.Child("Bio: " + record.Bio)
rootTree.Child("Site: " + record.Site)
rootTree.Child("Avatar: " + record.Avatar)
rootTree.Child(record.EmploymentTree())
rootTree.Child(record.FacebookTree())
rootTree.Child(record.GitHubTree())
rootTree.Child(record.TwitterTree())
rootTree.Child(record.LinkedInTree())
rootTree.Child(record.GooglePlusTree())
rootTree.Child(record.GravatarTree())
rootTree.Child("Fuzzy: " + fmt.Sprintf("%t", record.Fuzzy))
rootTree.Child("Email Provider: " + record.EmailProvider)
rootTree.Child("Indexed At: " + record.IndexedAt)
rootTree.Child("Phone: " + record.Phone)
rootTree.Child("Active At: " + record.ActiveAt)
rootTree.Child("Inactive At: " + record.InactiveAt)
// Styles
rootTree.Enumerator(tree.RoundedEnumerator)
rootTree.EnumeratorStyle(enumeratorStyle)
rootTree.RootStyle(rootStyle)
rootTree.ItemStyle(itemStyle)
// Print Tree
fmt.Println(rootTree)
}
func HunterCombinedEnrichmentTree(root string, record sqlite.CombinedData) {
enumeratorStyle := lipgloss.NewStyle().Foreground(purple).MarginRight(1)
rootStyle := lipgloss.NewStyle().Foreground(lipgloss.Color("63"))
itemStyle := lipgloss.NewStyle().Foreground(gray)
rootTree := tree.Root(root)
// Root Tree Children
rootTree.Child(personTree(record.Person))
rootTree.Child(companyTree(record.Company))
// Styles
rootTree.Enumerator(tree.RoundedEnumerator)
rootTree.EnumeratorStyle(enumeratorStyle)
rootTree.RootStyle(rootStyle)
rootTree.ItemStyle(itemStyle)
// Print Tree
fmt.Println(rootTree)
}
func companyTree(record sqlite.CompanyData) *tree.Tree {
companyTree := tree.Root("Company")
// Company Tree Children
companyTree.Child("ID: " + record.ID)
companyTree.Child("Name: " + record.Name)
companyTree.Child("Legal Name: " + record.LegalName)
companyTree.Child("Domain: " + record.Domain)
companyTree.Child(record.DomainAliasesTree())
companyTree.Child(record.SiteTree())
companyTree.Child(record.CategoryTree())
companyTree.Child(record.TagsTree())
companyTree.Child("Description: " + record.Description)
companyTree.Child("Founded Year: " + fmt.Sprintf("%d", record.FoundedYear))
companyTree.Child("Location: " + record.Location)
companyTree.Child("Time Zone: " + record.TimeZone)
companyTree.Child("UTC Offset: " + fmt.Sprintf("%d", record.UTCOffset))
companyTree.Child(record.GeoTree())
companyTree.Child("Logo: " + record.Logo)
companyTree.Child(record.FacebookTree())
companyTree.Child(record.LinkedInTree())
companyTree.Child(record.TwitterTree())
companyTree.Child(record.CrunchbaseTree())
companyTree.Child(record.YouTubeTree())
companyTree.Child("Email Provider: " + record.EmailProvider)
companyTree.Child("Type: " + record.Type)
companyTree.Child("Ticker: " + record.Ticker)
companyTree.Child(record.IdentifiersTree())
companyTree.Child("Phone: " + record.Phone)
companyTree.Child(record.MetricsTree())
companyTree.Child("Indexed At: " + record.IndexedAt)
companyTree.Child(record.TechTree())
companyTree.Child(record.TechCategoriesTree())
companyTree.Child(record.ParentTree())
companyTree.Child(record.UltimateParentTree())
return companyTree
}
func personTree(record sqlite.PersonData) *tree.Tree {
personTree := tree.Root("Person")
// Person Tree Children
personTree.Child("ID: " + record.ID)
personTree.Child(record.NameTree())
personTree.Child("Email: " + record.Email)
personTree.Child("Location: " + record.Location)
personTree.Child("Time Zone: " + record.TimeZone)
personTree.Child("UTC Offset: " + fmt.Sprintf("%d", record.UTCOffset))
personTree.Child(record.GeoTree())
personTree.Child("Bio: " + record.Bio)
personTree.Child("Site: " + record.Site)
personTree.Child("Avatar: " + record.Avatar)
personTree.Child(record.EmploymentTree())
personTree.Child(record.FacebookTree())
personTree.Child(record.GitHubTree())
personTree.Child(record.TwitterTree())
personTree.Child(record.LinkedInTree())
personTree.Child(record.GooglePlusTree())
personTree.Child(record.GravatarTree())
personTree.Child("Fuzzy: " + fmt.Sprintf("%t", record.Fuzzy))
personTree.Child("Email Provider: " + record.EmailProvider)
personTree.Child("Indexed At: " + record.IndexedAt)
personTree.Child("Phone: " + record.Phone)
personTree.Child("Active At: " + record.ActiveAt)
personTree.Child("Inactive At: " + record.InactiveAt)
return personTree
}
-419
View File
@@ -1,419 +0,0 @@
package sqlite
import (
"fmt"
"go.uber.org/zap"
"gorm.io/gorm"
"time"
)
// QueryResults queries the database for results based on the provided options
func QueryResults(options *DBOptions) ([]Result, error) {
db := GetDB()
var results []Result
query := db.Model(&Result{})
// Apply filters based on the provided options
query = applyFilters(query, options)
// Apply limit
if options.Limit > 0 {
query = query.Limit(options.Limit)
}
// Execute the query
if err := query.Find(&results).Error; err != nil {
zap.L().Error("query_results",
zap.String("message", "failed to query results"),
zap.Error(err),
)
return nil, fmt.Errorf("failed to query results: %w", err)
}
return results, nil
}
// applyFilters applies filters to the query based on the provided options
func applyFilters(query *gorm.DB, options *DBOptions) *gorm.DB {
// Helper function to apply filter based on exact match setting
applyFilter := func(field, value string) *gorm.DB {
if value == "" {
return query
}
if options.ExactMatch {
return query.Where(field+" = ?", value)
} else {
return query.Where(field+" LIKE ?", "%"+value+"%")
}
}
// Apply filters for each field if provided
if options.Email != "" {
query = applyFilter("email", options.Email)
}
if options.Username != "" {
query = applyFilter("username", options.Username)
}
if options.IPAddress != "" {
query = applyFilter("ip_address", options.IPAddress)
}
if options.Password != "" {
query = applyFilter("password", options.Password)
}
if options.HashedPassword != "" {
query = applyFilter("hashed_password", options.HashedPassword)
}
if options.Name != "" {
query = applyFilter("name", options.Name)
}
if options.Vin != "" {
query = applyFilter("vin", options.Vin)
}
if options.LicensePlate != "" {
query = applyFilter("license_plate", options.LicensePlate)
}
if options.Address != "" {
query = applyFilter("address", options.Address)
}
if options.Phone != "" {
query = applyFilter("phone", options.Phone)
}
if options.Social != "" {
query = applyFilter("social", options.Social)
}
if options.CryptoCurrencyAddress != "" {
query = applyFilter("cryptocurrency_address", options.CryptoCurrencyAddress)
}
if options.Domain != "" {
query = applyFilter("url", options.Domain)
}
// Apply non-empty field filters
for _, field := range options.NonEmptyFields {
switch field {
case "username":
query = query.Where("JSON_ARRAY_LENGTH(username) > 0")
case "email":
query = query.Where("JSON_ARRAY_LENGTH(email) > 0")
case "ip_address", "ipaddress", "ip":
query = query.Where("JSON_ARRAY_LENGTH(ip_address) > 0")
case "password":
query = query.Where("JSON_ARRAY_LENGTH(password) > 0")
case "hashed_password", "hash":
query = query.Where("JSON_ARRAY_LENGTH(hashed_password) > 0")
case "name":
query = query.Where("JSON_ARRAY_LENGTH(name) > 0")
case "vin":
query = query.Where("JSON_ARRAY_LENGTH(vin) > 0")
case "license_plate", "license":
query = query.Where("JSON_ARRAY_LENGTH(license_plate) > 0")
case "address":
query = query.Where("JSON_ARRAY_LENGTH(address) > 0")
case "phone":
query = query.Where("JSON_ARRAY_LENGTH(phone) > 0")
case "social":
query = query.Where("JSON_ARRAY_LENGTH(social) > 0")
case "cryptocurrency_address", "crypto":
query = query.Where("JSON_ARRAY_LENGTH(cryptocurrency_address) > 0")
case "url", "domain":
query = query.Where("JSON_ARRAY_LENGTH(url) > 0")
}
}
return query
}
// GetResultsCount returns the count of results matching the provided options
func GetResultsCount(options *DBOptions) (int64, error) {
db := GetDB()
var count int64
query := db.Model(&Result{})
// Apply filters based on the provided options
query = applyFilters(query, options)
// Count the results
if err := query.Count(&count).Error; err != nil {
zap.L().Error("get_results_count",
zap.String("message", "failed to count results"),
zap.Error(err),
)
return 0, fmt.Errorf("failed to count results: %w", err)
}
return count, nil
}
// QueryRuns queries the database for previous query runs (QueryOptions) based on the provided filters
func QueryRuns(limit, lastXRuns int, startDate, endDate time.Time, containsQuery string) ([]QueryOptions, error) {
db := GetDB()
var runs []QueryOptions
query := db.Model(&QueryOptions{})
// Apply date range filter if provided
if lastXRuns > 0 {
query = query.Order("created_at DESC").Limit(lastXRuns)
} else if !startDate.IsZero() && !endDate.IsZero() {
query = query.Where("created_at BETWEEN ? AND ?", startDate, endDate)
} else if !startDate.IsZero() {
query = query.Where("created_at >= ?", startDate)
} else if !endDate.IsZero() {
query = query.Where("created_at <= ?", endDate)
}
// Apply query filter if provided
if containsQuery != "" {
// SearchTerm in all query fields
query = query.Where(
"username_query LIKE ? OR "+
"email_query LIKE ? OR "+
"ip_query LIKE ? OR "+
"pass_query LIKE ? OR "+
"hash_query LIKE ? OR "+
"name_query LIKE ? OR "+
"domain_query LIKE ? OR "+
"vin_query LIKE ? OR "+
"license_plate_query LIKE ? OR "+
"address_query LIKE ? OR "+
"phone_query LIKE ? OR "+
"social_query LIKE ? OR "+
"crypto_address_query LIKE ?",
"%"+containsQuery+"%", "%"+containsQuery+"%", "%"+containsQuery+"%",
"%"+containsQuery+"%", "%"+containsQuery+"%", "%"+containsQuery+"%",
"%"+containsQuery+"%", "%"+containsQuery+"%", "%"+containsQuery+"%",
"%"+containsQuery+"%", "%"+containsQuery+"%", "%"+containsQuery+"%",
"%"+containsQuery+"%",
)
}
// Apply limit
if limit > 0 {
query = query.Limit(limit)
}
// Order by most recent first
query = query.Order("created_at DESC")
// Execute the query
if err := query.Find(&runs).Error; err != nil {
zap.L().Error("query_runs",
zap.String("message", "failed to query runs"),
zap.Error(err),
)
return nil, fmt.Errorf("failed to query runs: %w", err)
}
return runs, nil
}
// GetRunsCount returns the count of runs matching the provided filters
func GetRunsCount(lastXRuns int, startDate, endDate time.Time, containsQuery string) (int64, error) {
db := GetDB()
var count int64
query := db.Model(&QueryOptions{})
// Apply date range filter if provided
if lastXRuns > 0 {
query = query.Order("created_at DESC").Limit(lastXRuns)
} else if !startDate.IsZero() && !endDate.IsZero() {
query = query.Where("created_at BETWEEN ? AND ?", startDate, endDate)
} else if !startDate.IsZero() {
query = query.Where("created_at >= ?", startDate)
} else if !endDate.IsZero() {
query = query.Where("created_at <= ?", endDate)
}
// Apply query filter if provided
if containsQuery != "" {
// SearchTerm in all query fields
query = query.Where(
"username_query LIKE ? OR "+
"email_query LIKE ? OR "+
"ip_query LIKE ? OR "+
"pass_query LIKE ? OR "+
"hash_query LIKE ? OR "+
"name_query LIKE ? OR "+
"domain_query LIKE ? OR "+
"vin_query LIKE ? OR "+
"license_plate_query LIKE ? OR "+
"address_query LIKE ? OR "+
"phone_query LIKE ? OR "+
"social_query LIKE ? OR "+
"crypto_address_query LIKE ?",
"%"+containsQuery+"%", "%"+containsQuery+"%", "%"+containsQuery+"%",
"%"+containsQuery+"%", "%"+containsQuery+"%", "%"+containsQuery+"%",
"%"+containsQuery+"%", "%"+containsQuery+"%", "%"+containsQuery+"%",
"%"+containsQuery+"%", "%"+containsQuery+"%", "%"+containsQuery+"%",
"%"+containsQuery+"%",
)
}
// Count the results
if err := query.Count(&count).Error; err != nil {
zap.L().Error("get_runs_count",
zap.String("message", "failed to count runs"),
zap.Error(err),
)
return 0, fmt.Errorf("failed to count runs: %w", err)
}
return count, nil
}
// QueryCreds queries the database for credentials based on the provided filters
func QueryCreds(options *DBOptions) ([]Creds, error) {
db := GetDB()
var creds []Creds
query := db.Model(&Creds{})
// Apply filters based on the provided options
if options.Username != "" {
if options.ExactMatch {
query = query.Where("username = ?", options.Username)
} else {
query = query.Where("username LIKE ?", "%"+options.Username+"%")
}
}
if options.Email != "" {
if options.ExactMatch {
query = query.Where("email = ?", options.Email)
} else {
query = query.Where("email LIKE ?", "%"+options.Email+"%")
}
}
if options.Password != "" {
if options.ExactMatch {
query = query.Where("password = ?", options.Password)
} else {
query = query.Where("password LIKE ?", "%"+options.Password+"%")
}
}
// Apply limit
if options.Limit > 0 {
query = query.Limit(options.Limit)
}
// Execute the query
if err := query.Find(&creds).Error; err != nil {
zap.L().Error("query_creds",
zap.String("message", "failed to query credentials"),
zap.Error(err),
)
return nil, fmt.Errorf("failed to query credentials: %w", err)
}
return creds, nil
}
// GetCredsCount returns the count of credentials matching the provided filters
func GetCredsCount(options *DBOptions) (int64, error) {
db := GetDB()
var count int64
query := db.Model(&Creds{})
// Apply filters based on the provided options
if options.Username != "" {
if options.ExactMatch {
query = query.Where("username = ?", options.Username)
} else {
query = query.Where("username LIKE ?", "%"+options.Username+"%")
}
}
if options.Email != "" {
if options.ExactMatch {
query = query.Where("email = ?", options.Email)
} else {
query = query.Where("email LIKE ?", "%"+options.Email+"%")
}
}
if options.Password != "" {
if options.ExactMatch {
query = query.Where("password = ?", options.Password)
} else {
query = query.Where("password LIKE ?", "%"+options.Password+"%")
}
}
// Count the results
if err := query.Count(&count).Error; err != nil {
zap.L().Error("get_creds_count",
zap.String("message", "failed to count credentials"),
zap.Error(err),
)
return 0, fmt.Errorf("failed to count credentials: %w", err)
}
return count, nil
}
// ExecuteRawQuery executes a raw SQL query and returns the results as a slice of maps
func ExecuteRawQuery(query string) ([]map[string]interface{}, error) {
db := GetDB()
rows, err := db.Raw(query).Rows()
if err != nil {
zap.L().Error("raw_query",
zap.String("message", "failed to execute raw query"),
zap.Error(err),
)
return nil, fmt.Errorf("failed to execute raw query: %w", err)
}
defer rows.Close()
columns, err := rows.Columns()
if err != nil {
zap.L().Error("raw_query",
zap.String("message", "failed to get columns from raw query"),
zap.Error(err),
)
return nil, fmt.Errorf("failed to get columns from raw query: %w", err)
}
var results []map[string]interface{}
for rows.Next() {
// Create a slice of interface{} to hold the values
values := make([]interface{}, len(columns))
pointers := make([]interface{}, len(columns))
for i := range values {
pointers[i] = &values[i]
}
// Scan the result into the pointers
if err := rows.Scan(pointers...); err != nil {
zap.L().Error("raw_query",
zap.String("message", "failed to scan row from raw query"),
zap.Error(err),
)
return nil, fmt.Errorf("failed to scan row from raw query: %w", err)
}
// Create a map for this row
rowMap := make(map[string]interface{})
for i, col := range columns {
val := values[i]
rowMap[col] = val
}
results = append(results, rowMap)
}
return results, nil
}
+63 -1
View File
@@ -51,7 +51,8 @@ func InitDB(dbPath string) (*gorm.DB, error) {
}
// Auto migrate your models
err = db.AutoMigrate(&Result{}, &Creds{}, &QueryOptions{}, &Creds{}, &WhoisRecord{}, &SubdomainRecord{}, &HistoryRecord{}, &LookupResult{})
err = db.AutoMigrate(&Result{}, &Creds{}, &QueryOptions{}, &Creds{}, &WhoisRecord{}, &SubdomainRecord{},
&HistoryRecord{}, &LookupResult{}, &HunterDomainData{}, &HunterEmail{}, &PersonData{})
if err != nil {
zap.L().Error("Failed to migrate database", zap.Error(err))
return nil, fmt.Errorf("failed to migrate database: %w", err)
@@ -255,3 +256,64 @@ func StoreIPLookup(ipLookup []LookupResult) error {
return lastErr
}
func StoreHunterDomain(hunterDomain HunterDomainData) error {
db := GetDB()
// Use OnConflict clause to handle duplicates
err := db.Clauses(clause.OnConflict{DoNothing: true}).Create(&hunterDomain).Error
if err != nil {
zap.L().Error("store_hunter_domain",
zap.String("message", "failed to store hunter domain"),
zap.Error(err))
return err
}
return nil
}
func StoreHunterEmails(hunterEmails []HunterEmail) error {
if len(hunterEmails) == 0 {
return nil
}
zap.L().Info("Storing hunter emails", zap.Int("count", len(hunterEmails)))
db := GetDB()
// Use batch insert with conflict handling
const batchSize = 100
var lastErr error
for i := 0; i < len(hunterEmails); i += batchSize {
end := i + batchSize
if end > len(hunterEmails) {
end = len(hunterEmails)
}
batch := hunterEmails[i:end]
// Use Clauses with OnConflict DoNothing to skip conflicts
err := db.Clauses(clause.OnConflict{DoNothing: true}).CreateInBatches(&batch, batchSize).Error
if err != nil {
zap.L().Warn("Error storing some hunter emails", zap.Error(err))
lastErr = err
// Continue with next batch despite error
}
}
return lastErr
}
func StorePersonData(personData PersonData) error {
db := GetDB()
// Use OnConflict clause to handle duplicates
err := db.Clauses(clause.OnConflict{DoNothing: true}).Create(&personData).Error
if err != nil {
zap.L().Error("store_person_data",
zap.String("message", "failed to store person data"),
zap.Error(err))
return err
}
return nil
}
+751
View File
@@ -0,0 +1,751 @@
package sqlite
import (
"fmt"
"github.com/charmbracelet/lipgloss/tree"
"gorm.io/gorm"
)
// HunterDomainSearchResult represents the response from Hunter.io domain search API
type HunterDomainSearchResult struct {
Data HunterDomainData `json:"data" gorm:"embedded;embeddedPrefix:data_"`
Meta HunterMeta `json:"meta" gorm:"embedded;embeddedPrefix:meta_"`
}
// HunterDomainData contains the main domain information
type HunterDomainData struct {
gorm.Model
Domain string `json:"domain" gorm:"unique"`
Disposable bool `json:"disposable"`
Webmail bool `json:"webmail"`
AcceptAll bool `json:"accept_all"`
Pattern string `json:"pattern"`
Organization string `json:"organization"`
Description string `json:"description"`
Industry string `json:"industry"`
Twitter string `json:"twitter"`
Facebook string `json:"facebook"`
Linkedin string `json:"linkedin"`
Instagram string `json:"instagram"`
Youtube string `json:"youtube"`
Technologies []string `json:"technologies" gorm:"serializer:json"`
Country string `json:"country"`
State string `json:"state"`
City string `json:"city"`
PostalCode string `json:"postal_code"`
Street string `json:"street"`
Headcount string `json:"headcount"`
CompanyType string `json:"company_type"`
Emails []HunterEmail `json:"emails" gorm:"serializer:json"`
LinkedDomains []string `json:"linked_domains" gorm:"serializer:json"`
}
func (h *HunterDomainData) String() string {
return fmt.Sprintf("Domain: %s\nDisposable: %t\nWebmail: %t\nAcceptAll: %t\nPattern: %s\nOrganization: %s\nDescription: %s\nIndustry: %s\nTwitter: %s\nFacebook: %s\nLinkedin: %s\nInstagram: %s\nYoutube: %s\nTechnologies: %v\nCountry: %s\nState: %s\nCity: %s\nPostalCode: %s\nStreet: %s\nHeadcount: %s\nCompanyType: %s\nEmails: %v\nLinkedDomains: %v\n",
h.Domain, h.Disposable, h.Webmail, h.AcceptAll, h.Pattern, h.Organization, h.Description, h.Industry, h.Twitter, h.Facebook, h.Linkedin, h.Instagram, h.Youtube, h.Technologies, h.Country, h.State, h.City, h.PostalCode, h.Street, h.Headcount, h.CompanyType, h.Emails, h.LinkedDomains)
}
func (HunterDomainData) TableName() string {
return "hunter_domain"
}
// HunterEmail represents an email found for the domain
type HunterEmail struct {
gorm.Model
Domain string `json:"domain,omitempty"`
Value string `json:"value" gorm:"unique"`
Type string `json:"type"`
Confidence int `json:"confidence"`
Sources []HunterSource `json:"sources" gorm:"serializer:json"`
FirstName string `json:"first_name"`
LastName string `json:"last_name"`
Position string `json:"position"`
PositionRaw string `json:"position_raw"`
Seniority string `json:"seniority"`
Department string `json:"department"`
Linkedin string `json:"linkedin"`
Twitter string `json:"twitter"`
PhoneNumber string `json:"phone_number"`
Verification HunterVerification `json:"verification" gorm:"embedded;embeddedPrefix:verification_"`
}
func (he *HunterEmail) ToTree() *tree.Tree {
emailTree := tree.Root(he.Value)
emailTree.Child("Type: " + he.Type)
emailTree.Child("Confidence: " + fmt.Sprintf("%d", he.Confidence))
emailTree.Child("FirstName: " + he.FirstName)
emailTree.Child("LastName: " + he.LastName)
emailTree.Child("Position: " + he.Position)
emailTree.Child("PositionRaw: " + he.PositionRaw)
emailTree.Child("Seniority: " + he.Seniority)
emailTree.Child("Department: " + he.Department)
emailTree.Child("Linkedin: " + he.Linkedin)
emailTree.Child("Twitter: " + he.Twitter)
emailTree.Child("PhoneNumber: " + he.PhoneNumber)
emailTree.Child(he.Verification.ToTree())
return emailTree
}
func (he *HunterEmail) String() string {
return fmt.Sprintf("Value: %s\nType: %s\nConfidence: %d\nSources: %v\nFirstName: %s\nLastName: %s\nPosition: %s\nPositionRaw: %s\nSeniority: %s\nDepartment: %s\nLinkedin: %s\nTwitter: %s\nPhoneNumber: %s\nVerification: %v\n",
he.Value, he.Type, he.Confidence, he.Sources, he.FirstName, he.LastName, he.Position, he.PositionRaw, he.Seniority, he.Department, he.Linkedin, he.Twitter, he.PhoneNumber, he.Verification)
}
func (HunterEmail) TableName() string {
return "hunter_email"
}
// HunterSource represents where an email was found
type HunterSource struct {
Domain string `json:"domain"`
URI string `json:"uri"`
ExtractedOn string `json:"extracted_on"`
LastSeenOn string `json:"last_seen_on"`
StillOnPage bool `json:"still_on_page"`
}
// HunterVerification represents the verification status of an email
type HunterVerification struct {
Date string `json:"date"`
Status string `json:"status"`
}
func (hv *HunterVerification) ToTree() *tree.Tree {
verificationTree := tree.Root("Verification")
verificationTree.Child("Date: " + hv.Date)
verificationTree.Child("Status: " + hv.Status)
return verificationTree
}
// HunterMeta contains metadata about the API response
type HunterMeta struct {
Results int `json:"results"`
Limit int `json:"limit"`
Offset int `json:"offset"`
Params HunterSearchParams `json:"params" gorm:"embedded;embeddedPrefix:params_"`
}
// HunterSearchParams contains the parameters used in the search
type HunterSearchParams struct {
Domain string `json:"domain"`
Company string `json:"company"`
Type string `json:"type"`
Seniority string `json:"seniority"`
Department string `json:"department"`
}
// HunterEmailFinderResponse represents the response from Hunter.io email finder API
type HunterEmailFinderResponse struct {
Data HunterEmailFinderData `json:"data" gorm:"embedded;embeddedPrefix:data_"`
Meta EmailFinderMeta `json:"meta" gorm:"embedded;embeddedPrefix:meta_"`
}
// HunterEmailFinderData contains the main email information
type HunterEmailFinderData struct {
FirstName string `json:"first_name"`
LastName string `json:"last_name"`
Email string `json:"email"`
Score int `json:"score"`
Domain string `json:"domain"`
AcceptAll bool `json:"accept_all"`
Position string `json:"position"`
Twitter string `json:"twitter"`
LinkedinURL string `json:"linkedin_url"`
PhoneNumber string `json:"phone_number"`
Company string `json:"company"`
Sources []HunterSource `json:"sources" gorm:"serializer:json"`
Verification HunterVerification `json:"verification" gorm:"embedded;embeddedPrefix:verification_"`
}
func (he *HunterEmailFinderData) String() string {
return fmt.Sprintf("FirstName: %s\nLastName: %s\nEmail: %s\nScore: %d\nDomain: %s\nAcceptAll: %t\nPosition: %s\nTwitter: %s\nLinkedinURL: %s\nPhoneNumber: %s\nCompany: %s\nSources: %v\nVerification: %v\n",
he.FirstName, he.LastName, he.Email, he.Score, he.Domain, he.AcceptAll, he.Position, he.Twitter, he.LinkedinURL, he.PhoneNumber, he.Company, he.Sources, he.Verification)
}
// EmailFinderMeta contains metadata about the API response
type EmailFinderMeta struct {
Params EmailFinderParams `json:"params" gorm:"embedded;embeddedPrefix:params_"`
}
// EmailFinderParams contains the parameters used in the search
type EmailFinderParams struct {
FirstName string `json:"first_name"`
LastName string `json:"last_name"`
FullName string `json:"full_name"`
Domain string `json:"domain"`
Company string `json:"company"`
MaxDuration string `json:"max_duration"`
}
func (HunterEmailFinderResponse) TableName() string {
return "hunter_email_finder"
}
// HunterEmailVerifyResponse represents the response from Hunter.io email verification API
type HunterEmailVerifyResponse struct {
Data HunterEmailVerifyData `json:"data" gorm:"embedded;embeddedPrefix:data_"`
Meta EmailVerifyMeta `json:"meta" gorm:"embedded;embeddedPrefix:meta_"`
}
// HunterEmailVerifyData contains the email verification information
type HunterEmailVerifyData struct {
Status string `json:"status"`
Result string `json:"result"`
DeprecationNotice string `json:"_deprecation_notice"`
Score int `json:"score"`
Email string `json:"email"`
Regexp bool `json:"regexp"`
Gibberish bool `json:"gibberish"`
Disposable bool `json:"disposable"`
Webmail bool `json:"webmail"`
MXRecords bool `json:"mx_records"`
SMTPServer bool `json:"smtp_server"`
SMTPCheck bool `json:"smtp_check"`
AcceptAll bool `json:"accept_all"`
Block bool `json:"block"`
Sources []HunterSource `json:"sources" gorm:"serializer:json"`
}
func (ev *HunterEmailVerifyData) String() string {
return fmt.Sprintf("Status: %s\nResult: %s\nDeprecationNotice: %s\nScore: %d\nEmail: %s\nRegexp: %t\nGibberish: %t\nDisposable: %t\nWebmail: %t\nMXRecords: %t\nSMTPServer: %t\nSMTPCheck: %t\nAcceptAll: %t\nBlock: %t\nSources: %v\n",
ev.Status, ev.Result, ev.DeprecationNotice, ev.Score, ev.Email, ev.Regexp, ev.Gibberish, ev.Disposable, ev.Webmail, ev.MXRecords, ev.SMTPServer, ev.SMTPCheck, ev.AcceptAll, ev.Block, ev.Sources)
}
// EmailVerifyMeta contains metadata about the API response
type EmailVerifyMeta struct {
Params EmailVerifyParams `json:"params" gorm:"embedded;embeddedPrefix:params_"`
}
// EmailVerifyParams contains the parameters used in the verification
type EmailVerifyParams struct {
Email string `json:"email"`
}
// HunterCompanyEnrichmentResponse represents the response from Hunter.io company enrichment API
type HunterCompanyEnrichmentResponse struct {
Data CompanyData `json:"data" gorm:"embedded;embeddedPrefix:data_"`
Meta CompanyMeta `json:"meta" gorm:"embedded;embeddedPrefix:meta_"`
}
// CompanyData contains the detailed company information
type CompanyData struct {
ID string `json:"id"`
Name string `json:"name"`
LegalName string `json:"legalName"`
Domain string `json:"domain"`
DomainAliases []string `json:"domainAliases" gorm:"serializer:json"`
Site CompanySite `json:"site" gorm:"embedded;embeddedPrefix:site_"`
Category Category `json:"category" gorm:"embedded;embeddedPrefix:category_"`
Tags []string `json:"tags" gorm:"serializer:json"`
Description string `json:"description"`
FoundedYear int `json:"foundedYear"`
Location string `json:"location"`
TimeZone string `json:"timeZone"`
UTCOffset int `json:"utcOffset"`
Geo Geography `json:"geo" gorm:"embedded;embeddedPrefix:geo_"`
Logo string `json:"logo"`
Facebook Facebook `json:"facebook" gorm:"embedded;embeddedPrefix:facebook_"`
LinkedIn LinkedIn `json:"linkedin" gorm:"embedded;embeddedPrefix:linkedin_"`
Twitter Twitter `json:"twitter" gorm:"embedded;embeddedPrefix:twitter_"`
Crunchbase Crunchbase `json:"crunchbase" gorm:"embedded;embeddedPrefix:crunchbase_"`
YouTube YouTube `json:"youtube" gorm:"embedded;embeddedPrefix:youtube_"`
EmailProvider string `json:"emailProvider"`
Type string `json:"type"`
Ticker string `json:"ticker"`
Identifiers Identifiers `json:"identifiers" gorm:"embedded;embeddedPrefix:identifiers_"`
Phone string `json:"phone"`
Metrics Metrics `json:"metrics" gorm:"embedded;embeddedPrefix:metrics_"`
IndexedAt string `json:"indexedAt"`
Tech []string `json:"tech" gorm:"serializer:json"`
TechCategories []string `json:"techCategories" gorm:"serializer:json"`
Parent ParentCompany `json:"parent" gorm:"embedded;embeddedPrefix:parent_"`
UltimateParent ParentCompany `json:"ultimateParent" gorm:"embedded;embeddedPrefix:ultimate_parent_"`
}
func (cd *CompanyData) String() string {
return fmt.Sprintf("ID: %s\nName: %s\nLegalName: %s\nDomain: %s\nDomainAliases: %v\nSite: %v\nCategory: %v\nTags: %v\nDescription: %s\nFoundedYear: %d\nLocation: %s\nTimeZone: %s\nUTCOffset: %d\nGeo: %v\nLogo: %s\nFacebook: %v\nLinkedIn: %v\nTwitter: %v\nCrunchbase: %v\nYouTube: %v\nEmailProvider: %s\nType: %s\nTicker: %s\nIdentifiers: %v\nPhone: %s\nMetrics: %v\nIndexedAt: %s\nTech: %v\nTechCategories: %v\nParent: %v\nUltimateParent: %v\n",
cd.ID, cd.Name, cd.LegalName, cd.Domain, cd.DomainAliases, cd.Site, cd.Category, cd.Tags, cd.Description, cd.FoundedYear, cd.Location, cd.TimeZone, cd.UTCOffset, cd.Geo, cd.Logo, cd.Facebook, cd.LinkedIn, cd.Twitter, cd.Crunchbase, cd.YouTube, cd.EmailProvider, cd.Type, cd.Ticker, cd.Identifiers, cd.Phone, cd.Metrics, cd.IndexedAt, cd.Tech, cd.TechCategories, cd.Parent, cd.UltimateParent)
}
func (cd *CompanyData) DomainAliasesTree() *tree.Tree {
domainAliasesTree := tree.Root("Domain Aliases")
for _, domainAlias := range cd.DomainAliases {
domainAliasesTree.Child(domainAlias)
}
return domainAliasesTree
}
func (cd *CompanyData) SiteTree() *tree.Tree {
siteTree := tree.Root("Site")
phoneTree := tree.Root("Phone Numbers")
for _, phoneNumber := range cd.Site.PhoneNumbers {
phoneTree.Child(phoneNumber)
}
emailTree := tree.Root("Email Addresses")
for _, emailAddress := range cd.Site.EmailAddresses {
emailTree.Child(emailAddress)
}
siteTree.Child(phoneTree)
siteTree.Child(emailTree)
return siteTree
}
func (cd *CompanyData) CategoryTree() *tree.Tree {
categoryTree := tree.Root("Category")
categoryTree.Child("Sector: " + cd.Category.Sector)
categoryTree.Child("Industry Group: " + cd.Category.IndustryGroup)
categoryTree.Child("Industry: " + cd.Category.Industry)
categoryTree.Child("Sub Industry: " + cd.Category.SubIndustry)
categoryTree.Child("GICS Code: " + cd.Category.GICSCode)
categoryTree.Child("SIC Code: " + cd.Category.SICCode)
sic4CodesTree := tree.Root("SIC 4 Codes")
for _, sic4Code := range cd.Category.SIC4Codes {
sic4CodesTree.Child(sic4Code)
}
categoryTree.Child(sic4CodesTree)
categoryTree.Child("NAICS Code: " + cd.Category.NAICSCode)
naics6CodesTree := tree.Root("NAICS 6 Codes")
for _, naics6Code := range cd.Category.NAICS6Codes {
naics6CodesTree.Child(naics6Code)
}
categoryTree.Child(naics6CodesTree)
naics6Codes2022Tree := tree.Root("NAICS 6 Codes 2022")
for _, naics6Code2022 := range cd.Category.NAICS6Codes2022 {
naics6Codes2022Tree.Child(naics6Code2022)
}
categoryTree.Child(naics6Codes2022Tree)
return categoryTree
}
func (cd *CompanyData) GeoTree() *tree.Tree {
geoTree := tree.Root("Geo")
geoTree.Child("Street Number: " + cd.Geo.StreetNumber)
geoTree.Child("Street Name: " + cd.Geo.StreetName)
geoTree.Child("Sub Premise: " + cd.Geo.SubPremise)
geoTree.Child("Street Address: " + cd.Geo.StreetAddress)
geoTree.Child("City: " + cd.Geo.City)
geoTree.Child("Postal Code: " + cd.Geo.PostalCode)
geoTree.Child("State: " + cd.Geo.State)
geoTree.Child("State Code: " + cd.Geo.StateCode)
geoTree.Child("Country: " + cd.Geo.Country)
geoTree.Child("Country Code: " + cd.Geo.CountryCode)
geoTree.Child("Latitude: " + fmt.Sprintf("%f", cd.Geo.Lat))
geoTree.Child("Longitude: " + fmt.Sprintf("%f", cd.Geo.Lng))
return geoTree
}
func (cd *CompanyData) FacebookTree() *tree.Tree {
facebookTree := tree.Root("Facebook")
facebookTree.Child("Handle: " + cd.Facebook.Handle)
facebookTree.Child("Likes: " + fmt.Sprintf("%d", cd.Facebook.Likes))
return facebookTree
}
func (cd *CompanyData) LinkedInTree() *tree.Tree {
linkedinTree := tree.Root("LinkedIn")
linkedinTree.Child("Handle: " + cd.LinkedIn.Handle)
return linkedinTree
}
func (cd *CompanyData) TwitterTree() *tree.Tree {
twitterTree := tree.Root("Twitter")
twitterTree.Child("Handle: " + cd.Twitter.Handle)
twitterTree.Child("ID: " + cd.Twitter.ID)
twitterTree.Child("Bio: " + cd.Twitter.Bio)
twitterTree.Child("Followers: " + fmt.Sprintf("%d", cd.Twitter.Followers))
twitterTree.Child("Following: " + fmt.Sprintf("%d", cd.Twitter.Following))
twitterTree.Child("Location: " + cd.Twitter.Location)
twitterTree.Child("Site: " + cd.Twitter.Site)
twitterTree.Child("Avatar" + cd.Twitter.Avatar)
return twitterTree
}
func (cd *CompanyData) CrunchbaseTree() *tree.Tree {
crunchbaseTree := tree.Root("Crunchbase")
crunchbaseTree.Child("Handle: " + cd.Crunchbase.Handle)
return crunchbaseTree
}
func (cd *CompanyData) YouTubeTree() *tree.Tree {
youtubeTree := tree.Root("YouTube")
youtubeTree.Child("Handle: " + cd.YouTube.Handle)
return youtubeTree
}
func (cd *CompanyData) IdentifiersTree() *tree.Tree {
identifiersTree := tree.Root("Identifiers")
identifiersTree.Child("UsEIN: " + cd.Identifiers.UsEIN)
return identifiersTree
}
func (cd *CompanyData) MetricsTree() *tree.Tree {
metricsTree := tree.Root("Metrics")
metricsTree.Child("Alexa Us Rank: " + fmt.Sprintf("%d", cd.Metrics.AlexaUsRank))
metricsTree.Child("Alexa Global Rank: " + fmt.Sprintf("%d", cd.Metrics.AlexaGlobalRank))
metricsTree.Child("Traffic Rank: " + cd.Metrics.TrafficRank)
metricsTree.Child("Employees: " + cd.Metrics.Employees)
metricsTree.Child("Market Cap: " + cd.Metrics.MarketCap)
metricsTree.Child("Raised: " + cd.Metrics.Raised)
metricsTree.Child("Annual Revenue: " + cd.Metrics.AnnualRevenue)
metricsTree.Child("Estimated Annual Revenue: " + cd.Metrics.EstimatedAnnualRevenue)
metricsTree.Child("Fiscal Year End: " + cd.Metrics.FiscalYearEnd)
return metricsTree
}
func (cd *CompanyData) TagsTree() *tree.Tree {
tagsTree := tree.Root("Tags")
for _, tag := range cd.Tags {
tagsTree.Child(tag)
}
return tagsTree
}
func (cd *CompanyData) TechTree() *tree.Tree {
techTree := tree.Root("Tech")
for _, tech := range cd.Tech {
techTree.Child(tech)
}
return techTree
}
func (cd *CompanyData) TechCategoriesTree() *tree.Tree {
techCategoriesTree := tree.Root("Tech Categories")
for _, techCategory := range cd.TechCategories {
techCategoriesTree.Child(techCategory)
}
return techCategoriesTree
}
func (cd *CompanyData) ParentTree() *tree.Tree {
parentTree := tree.Root("Parent")
parentTree.Child("Domain: " + cd.Parent.Domain)
return parentTree
}
func (cd *CompanyData) UltimateParentTree() *tree.Tree {
ultimateParentTree := tree.Root("Ultimate Parent")
ultimateParentTree.Child("Domain: " + cd.UltimateParent.Domain)
return ultimateParentTree
}
// CompanySite contains contact information from the company website
type CompanySite struct {
PhoneNumbers []string `json:"phoneNumbers" gorm:"serializer:json"`
EmailAddresses []string `json:"emailAddresses" gorm:"serializer:json"`
}
// Category contains industry classification information
type Category struct {
Sector string `json:"sector"`
IndustryGroup string `json:"industryGroup"`
Industry string `json:"industry"`
SubIndustry string `json:"subIndustry"`
GICSCode string `json:"gicsCode"`
SICCode string `json:"sicCode"`
SIC4Codes []string `json:"sic4Codes" gorm:"serializer:json"`
NAICSCode string `json:"naicsCode"`
NAICS6Codes []string `json:"naics6Codes" gorm:"serializer:json"`
NAICS6Codes2022 []string `json:"naics6Codes2022" gorm:"serializer:json"`
}
// Geography contains location information
type Geography struct {
StreetNumber string `json:"streetNumber"`
StreetName string `json:"streetName"`
SubPremise string `json:"subPremise"`
StreetAddress string `json:"streetAddress"`
City string `json:"city"`
PostalCode string `json:"postalCode"`
State string `json:"state"`
StateCode string `json:"stateCode"`
Country string `json:"country"`
CountryCode string `json:"countryCode"`
Lat float64 `json:"lat"`
Lng float64 `json:"lng"`
}
// Identifiers contains company identification numbers
type Identifiers struct {
UsEIN string `json:"usEIN"`
}
// Metrics contains company performance metrics
type Metrics struct {
AlexaUsRank int `json:"alexaUsRank"`
AlexaGlobalRank int `json:"alexaGlobalRank"`
TrafficRank string `json:"trafficRank"`
Employees string `json:"employees"`
MarketCap string `json:"marketCap"`
Raised string `json:"raised"`
AnnualRevenue string `json:"annualRevenue"`
EstimatedAnnualRevenue string `json:"estimatedAnnualRevenue"`
FiscalYearEnd string `json:"fiscalYearEnd"`
}
// ParentCompany contains information about parent companies
type ParentCompany struct {
Domain string `json:"domain"`
}
// CompanyMeta contains metadata about the API response
type CompanyMeta struct {
Domain string `json:"domain"`
}
func (HunterCompanyEnrichmentResponse) TableName() string {
return "hunter_company_enrichment"
}
// HunterPersonEnrichmentResponse represents the response from Hunter.io person enrichment API
type HunterPersonEnrichmentResponse struct {
Data PersonData `json:"data" gorm:"embedded;embeddedPrefix:data_"`
Meta PersonMeta `json:"meta" gorm:"embedded;embeddedPrefix:meta_"`
}
// PersonData contains the detailed person information
type PersonData struct {
gorm.Model
ID string `json:"id"`
Name PersonName `json:"name" gorm:"embedded;embeddedPrefix:name_"`
Email string `json:"email" gorm:"unique"`
Location string `json:"location"`
TimeZone string `json:"timeZone"`
UTCOffset int `json:"utcOffset"`
Geo PersonGeo `json:"geo" gorm:"embedded;embeddedPrefix:geo_"`
Bio string `json:"bio"`
Site string `json:"site"`
Avatar string `json:"avatar"`
Employment Employment `json:"employment" gorm:"embedded;embeddedPrefix:employment_"`
Facebook Facebook `json:"facebook" gorm:"embedded;embeddedPrefix:facebook_"`
GitHub GitHub `json:"github" gorm:"embedded;embeddedPrefix:github_"`
Twitter Twitter `json:"twitter" gorm:"embedded;embeddedPrefix:twitter_"`
LinkedIn LinkedIn `json:"linkedin" gorm:"embedded;embeddedPrefix:linkedin_"`
GooglePlus GooglePlus `json:"googleplus" gorm:"embedded;embeddedPrefix:googleplus_"`
Gravatar Gravatar `json:"gravatar" gorm:"embedded;embeddedPrefix:gravatar_"`
Fuzzy bool `json:"fuzzy"`
EmailProvider string `json:"emailProvider"`
IndexedAt string `json:"indexedAt"`
Phone string `json:"phone"`
ActiveAt string `json:"activeAt"`
InactiveAt string `json:"inactiveAt"`
}
func (pd *PersonData) String() string {
return fmt.Sprintf("ID: %s\nName: %v\nEmail: %s\nLocation: %s\nTimeZone: %s\nUTCOffset: %d\nGeo: %v\nBio: %s\nSite: %s\nAvatar: %s\nEmployment: %v\nFacebook: %v\nGitHub: %v\nTwitter: %v\nLinkedIn: %v\nGooglePlus: %v\nGravatar: %v\nFuzzy: %t\nEmailProvider: %s\nIndexedAt: %s\nPhone: %s\nActiveAt: %s\nInactiveAt: %s\n",
pd.ID, pd.Name, pd.Email, pd.Location, pd.TimeZone, pd.UTCOffset, pd.Geo, pd.Bio, pd.Site, pd.Avatar, pd.Employment, pd.Facebook, pd.GitHub, pd.Twitter, pd.LinkedIn, pd.GooglePlus, pd.Gravatar, pd.Fuzzy, pd.EmailProvider, pd.IndexedAt, pd.Phone, pd.ActiveAt, pd.InactiveAt)
}
func (pd *PersonData) NameTree() *tree.Tree {
nameTree := tree.Root("Name")
nameTree.Child("Full Name: " + pd.Name.FullName)
nameTree.Child("Given Name: " + pd.Name.GivenName)
nameTree.Child("Family Name: " + pd.Name.FamilyName)
return nameTree
}
func (pd *PersonData) GeoTree() *tree.Tree {
geoTree := tree.Root("Geo")
geoTree.Child("City: " + pd.Geo.City)
geoTree.Child("State: " + pd.Geo.State)
geoTree.Child("State Code: " + pd.Geo.StateCode)
geoTree.Child("Country: " + pd.Geo.Country)
geoTree.Child("Country Code: " + pd.Geo.CountryCode)
geoTree.Child("Latitude: " + fmt.Sprintf("%f", pd.Geo.Lat))
geoTree.Child("Longitude: " + fmt.Sprintf("%f", pd.Geo.Lng))
return geoTree
}
func (pd *PersonData) EmploymentTree() *tree.Tree {
employmentTree := tree.Root("Employment")
employmentTree.Child("Domain: " + pd.Employment.Domain)
employmentTree.Child("Name: " + pd.Employment.Name)
employmentTree.Child("Title: " + pd.Employment.Title)
employmentTree.Child("Role: " + pd.Employment.Role)
employmentTree.Child("Sub Role: " + pd.Employment.SubRole)
employmentTree.Child("Seniority: " + pd.Employment.Seniority)
return employmentTree
}
func (pd *PersonData) FacebookTree() *tree.Tree {
facebookTree := tree.Root("Facebook")
facebookTree.Child("Handle: " + pd.Facebook.Handle)
facebookTree.Child("Likes: " + fmt.Sprintf("%d", pd.Facebook.Likes))
return facebookTree
}
func (pd *PersonData) GitHubTree() *tree.Tree {
githubTree := tree.Root("GitHub")
githubTree.Child("Handle: " + pd.GitHub.Handle)
githubTree.Child("ID: " + pd.GitHub.ID)
githubTree.Child("Avatar: " + pd.GitHub.Avatar)
githubTree.Child("Company: " + pd.GitHub.Company)
githubTree.Child("Blog: " + pd.GitHub.Blog)
githubTree.Child("Followers: " + fmt.Sprintf("%d", pd.GitHub.Followers))
githubTree.Child("Following: " + fmt.Sprintf("%d", pd.GitHub.Following))
return githubTree
}
func (pd *PersonData) TwitterTree() *tree.Tree {
twitterTree := tree.Root("Twitter")
twitterTree.Child("Handle: " + pd.Twitter.Handle)
twitterTree.Child("ID: " + pd.Twitter.ID)
twitterTree.Child("Bio: " + pd.Twitter.Bio)
twitterTree.Child("Followers: " + fmt.Sprintf("%d", pd.Twitter.Followers))
twitterTree.Child("Following: " + fmt.Sprintf("%d", pd.Twitter.Following))
twitterTree.Child("Location: " + pd.Twitter.Location)
twitterTree.Child("Site: " + pd.Twitter.Site)
twitterTree.Child("Avatar: " + pd.Twitter.Avatar)
return twitterTree
}
func (pd *PersonData) LinkedInTree() *tree.Tree {
linkedinTree := tree.Root("LinkedIn")
linkedinTree.Child("Handle: " + pd.LinkedIn.Handle)
return linkedinTree
}
func (pd *PersonData) GooglePlusTree() *tree.Tree {
googlePlusTree := tree.Root("GooglePlus")
googlePlusTree.Child("Handle: " + pd.GooglePlus.Handle)
return googlePlusTree
}
func (pd *PersonData) GravatarTree() *tree.Tree {
gravatarTree := tree.Root("Gravatar")
gravatarTree.Child("Handle: " + pd.Gravatar.Handle)
gravatarTree.Child("Avatar: " + pd.Gravatar.Avatar)
return gravatarTree
}
func (PersonData) TableName() string {
return "person"
}
// PersonName contains the person's name components
type PersonName struct {
FullName string `json:"fullName"`
GivenName string `json:"givenName"`
FamilyName string `json:"familyName"`
}
// PersonGeo contains location information for a person
type PersonGeo struct {
City string `json:"city"`
State string `json:"state"`
StateCode string `json:"stateCode"`
Country string `json:"country"`
CountryCode string `json:"countryCode"`
Lat float64 `json:"lat"`
Lng float64 `json:"lng"`
}
// Employment contains employment information
type Employment struct {
Domain string `json:"domain"`
Name string `json:"name"`
Title string `json:"title"`
Role string `json:"role"`
SubRole string `json:"subRole"`
Seniority string `json:"seniority"`
}
// GitHub contains GitHub profile information
type GitHub struct {
gorm.Model
Handle string `json:"handle"`
ID string `json:"id"`
Avatar string `json:"avatar"`
Company string `json:"company"`
Blog string `json:"blog"`
Followers int `json:"followers"`
Following int `json:"following"`
PersonID int `json:"person_id,omitempty"`
}
// GooglePlus contains Google+ profile information
type GooglePlus struct {
Handle string `json:"handle"`
PersonID int `json:"person_id,omitempty"`
}
// Gravatar contains Gravatar profile information
type Gravatar struct {
gorm.Model
Handle string `json:"handle"`
URLs []string `json:"urls" gorm:"serializer:json"`
Avatar string `json:"avatar"`
Avatars []string `json:"avatars" gorm:"serializer:json"`
PersonID int `json:"person_id,omitempty"`
}
// Facebook contains Facebook profile information
type Facebook struct {
Handle string `json:"handle"`
Likes int `json:"likes"`
}
// LinkedIn contains LinkedIn profile information
type LinkedIn struct {
Handle string `json:"handle"`
}
// Twitter contains Twitter profile information
type Twitter struct {
Handle string `json:"handle"`
ID string `json:"id"`
Bio string `json:"bio"`
Followers int `json:"followers"`
Following int `json:"following"`
Location string `json:"location"`
Site string `json:"site"`
Avatar string `json:"avatar"`
}
// Crunchbase contains Crunchbase profile information
type Crunchbase struct {
Handle string `json:"handle"`
}
// YouTube contains YouTube profile information
type YouTube struct {
Handle string `json:"handle"`
}
// PersonMeta contains metadata about the API response
type PersonMeta struct {
Email string `json:"email"`
}
// HunterCombinedEnrichmentResponse represents the response from Hunter.io combined enrichment API
type HunterCombinedEnrichmentResponse struct {
Data CombinedData `json:"data" gorm:"embedded;embeddedPrefix:data_"`
Meta CombinedMeta `json:"meta" gorm:"embedded;embeddedPrefix:meta_"`
}
// CombinedData contains both person and company information
type CombinedData struct {
Person PersonData `json:"person" gorm:"embedded;embeddedPrefix:person_"`
Company CompanyData `json:"company" gorm:"embedded;embeddedPrefix:company_"`
}
func (cbd *CombinedData) String() string {
return fmt.Sprintf("Person: %s\nCompany: %s",
cbd.Person.String(),
cbd.Company.String())
}
// CombinedMeta contains metadata about the API response
type CombinedMeta struct {
Email string `json:"email"`
}
// String returns a string representation of the combined enrichment response
func (c *HunterCombinedEnrichmentResponse) String() string {
return fmt.Sprintf("Person:\n%s\n\nCompany:\n%s",
c.Data.Person.String(),
c.Data.Company.String())
}
+15
View File
@@ -11,6 +11,9 @@ const (
WhoIsTable
SubdomainsTable
HistoryTable
LookupTable
HunterDomainTable
HunterEmailTable
UnknownTable
)
@@ -28,6 +31,12 @@ func GetTable(userInput string) Table {
return SubdomainsTable
case "history":
return HistoryTable
case "lookup":
return LookupTable
case "hunter_domain":
return HunterDomainTable
case "hunter_email":
return HunterEmailTable
default:
return UnknownTable
}
@@ -47,6 +56,12 @@ func (t Table) Object() interface{} {
return SubdomainRecord{}
case HistoryTable:
return HistoryRecord{}
case LookupTable:
return LookupResult{}
case HunterDomainTable:
return HunterDomainData{}
case HunterEmailTable:
return HunterEmail{}
default:
return nil
}
+2 -2
View File
@@ -50,7 +50,7 @@ func (w WhoisRecord) String() string {
sb.WriteString(fmt.Sprintf("Domain Name Ext: %s\n", w.DomainNameExt))
sb.WriteString(fmt.Sprintf("Registrar Name: %s\n", w.RegistrarName))
sb.WriteString(fmt.Sprintf("Registrar IANA ID: %s\n", w.RegistrarIANAID))
sb.WriteString(fmt.Sprintf("Contact Email: %s\n", w.ContactEmail))
sb.WriteString(fmt.Sprintf("Contact HunterEmail: %s\n", w.ContactEmail))
sb.WriteString(fmt.Sprintf("Estimated Domain Age: %d days\n", w.EstimatedDomainAge))
// Dates
@@ -379,7 +379,7 @@ func formatContact(sb *strings.Builder, contact ContactInfo, indent string) {
sb.WriteString(indent + "Organization: " + contact.Organization + "\n")
}
if contact.Email != "" {
sb.WriteString(indent + "Email: " + contact.Email + "\n")
sb.WriteString(indent + "HunterEmail: " + contact.Email + "\n")
}
if contact.Street != "" {
sb.WriteString(indent + "Street: " + contact.Street + "\n")