Removing Validation from hblink.py master


Cort N0MJS <n0mjs@...>
 

Removing source validation from hblink.py master probably won’t work because of hblink.py master’s a ability to repeat back to a group of connected clients. These two features will have to be separated. Why? Because without source validation, there’s nothing to stop it from repeating the traffic it just received back to the source.

So, first question for the group:

What’s more important, the ability for a single hblink.py master to “network” a group of repeaters tied to it, or the elimination of source validation?

--
Cort Buffington
H: +1-785-813-1501
M: +1-785-865-7206


Steve N4IRS
 

For me, hands down, ability for a single hblink.py master to “network” a group of repeaters tied to it.

On 07/28/2018 08:44 AM, Cort N0MJS via Groups.Io wrote:
Removing source validation from hblink.py master probably won’t work because of hblink.py master’s a ability to repeat back to a group of connected clients. These two features will have to be separated. Why? Because without source validation, there’s nothing to stop it from repeating the traffic it just received back to the source.

So, first question for the group:

What’s more important, the ability for a single hblink.py master to “network” a group of repeaters tied to it, or the elimination of source validation?

--
Cort Buffington
H: +1-785-813-1501
M: +1-785-865-7206







Matthew 2E0SIP
 

To clarify, does this refer to removing the validation for the source "RADIO_ID" sent in a frame from client -> master?

If yes, I gave this some thought a while ago, and the conclusion I came to was the source validation doesn't need to be removed parse, but instead performed on the Source IP and Port rather than the Radio ID.

So instead of having a dictionary of RADIO_ID keys and their associated parameters and states, you have a dictionary of sockets. You should never have more than one client connecting from the same combination of port and IP address, so I think this should work.

I didn't get around to implementing this, so I could well be missing something.... Also, if hblink.py is adjusted to re-write the RADIO_ID to itself when sending to a master, the above is a moot point.

Thanks all
Matthew


Cort N0MJS <n0mjs@...>
 


To clarify, does this refer to removing the validation for the source "RADIO_ID" sent in a frame from client -> master


Yes, client side validation was made option with loose (thanks again for that)

If yes, I gave this some thought a while ago, and the conclusion I came to was the source validation doesn't need to be removed parse, but instead performed on the Source IP and Port rather than the Radio ID.
So instead of having a dictionary of RADIO_ID keys and their associated parameters and states, you have a dictionary of sockets. You should never have more than one client connecting from the same combination of port and IP address, so I think this should work.

You’re now using network layer information to authenticate the application layer. Yes, it will work. I’m reluctant because that breaks the ISO model. I know a lot (most? all?) don’t care about that, but I’m usually a bit of a stickler for following as many rules as possible.

I’m working on convincing myself this is ok.

I didn't get around to implementing this, so I could well be missing something.... Also, if hblink.py is adjusted to re-write the RADIO_ID to itself when sending to a master, the above is a moot point.

And it used to do that, and someone complained that a particular scenario didn’t work right. So that line is actually still in there and commented out.


Perhaps the solution is to make it option in the system configuration for the master? That I would be pretty happy to implement, even if a stop-gap.


Thanks all
Matthew


--
Cort Buffington
H: +1-785-813-1501
M: +1-785-865-7206






Matthew 2E0SIP
 

Hi Cort,

Understood on the muddling of OSI layers - I get where you're coming from.

I think the easiest work around in that case is an option to rewrite the RADIO_ID for frames being sent to a Master to hblink's own Client RADIO_ID, as you suggested. 

Cheers


Cort N0MJS <n0mjs@...>
 

Ok, I’m going to admit to having not looked at any of this for a long time. I've found a few discrepancies in what I’ve done over time

I have a question that needs answered by the stewards of HB Protocol:

Should the “repeater_id” field (bytes 11-14 of the DMRD), when a master is sending to a client, be the radio ID of the client, or the radio ID of the originating repeater? I’ve sorta treated it both ways and don’t apparently know the real answer.

What is that field really about? And is it the same in either direction? (client -> master and master -> client).


With that answered I can clean up this entire mess once and for all. I asked G4KLX a question about HBP once and the answer I got was read the (completely un-commended C++) source code of MMDVM and figure it out… Not helpful. Brandmeister does not approve of or want HBlink talking to their system… so not a good move either. Maybe the DMR+ guys are more help, but I don’t know them. Maybe one of you who is friendlier with BM or Jonathan can get us an answer?


On Jul 28, 2018, at 4:15 PM, Matthew 2E0SIP <groups.io@...> wrote:

Hi Cort,

Understood on the muddling of OSI layers - I get where you're coming from.

I think the easiest work around in that case is an option to rewrite the RADIO_ID for frames being sent to a Master to hblink's own Client RADIO_ID, as you suggested. 

Cheers


--
Cort Buffington
H: +1-785-813-1501
M: +1-785-865-7206






Cort N0MJS <n0mjs@...>
 

Brandmeister always hands me my own radio id in the “repeater id” field, bytes 11-14 (inclusive) in the DMRD…. If anyone has seen different, let me know soon. I’m about ready to normalize it all so that a master always sends the client’s ID to it in that field, and a client always uses it’s ID in transmissions to the master. This will fix a few discrepancies, and will be done in hblink.py so all applications will be normalized in the process.

Speak now please….

0x49 DE N0MJS

On Jul 28, 2018, at 8:58 PM, Cort N0MJS via Groups.Io <n0mjs@...> wrote:

Ok, I’m going to admit to having not looked at any of this for a long time. I've found a few discrepancies in what I’ve done over time

I have a question that needs answered by the stewards of HB Protocol:

Should the “repeater_id” field (bytes 11-14 of the DMRD), when a master is sending to a client, be the radio ID of the client, or the radio ID of the originating repeater? I’ve sorta treated it both ways and don’t apparently know the real answer.

What is that field really about? And is it the same in either direction? (client -> master and master -> client).


With that answered I can clean up this entire mess once and for all. I asked G4KLX a question about HBP once and the answer I got was read the (completely un-commended C++) source code of MMDVM and figure it out… Not helpful. Brandmeister does not approve of or want HBlink talking to their system… so not a good move either. Maybe the DMR+ guys are more help, but I don’t know them. Maybe one of you who is friendlier with BM or Jonathan can get us an answer?


On Jul 28, 2018, at 4:15 PM, Matthew 2E0SIP <groups.io@...> wrote:

Hi Cort,

Understood on the muddling of OSI layers - I get where you're coming from.

I think the easiest work around in that case is an option to rewrite the RADIO_ID for frames being sent to a Master to hblink's own Client RADIO_ID, as you suggested. 

Cheers


--
Cort Buffington
H: +1-785-813-1501
M: +1-785-865-7206






Cort Buffington
785-865-7206


Matthew 2E0SIP
 

Hi Cort,

I asked Brandmeister / DMR+ a very long time ago and they both said they do not validate the Repeater ID. They both send the Repeater ID of the connected client in messages MST -> RPT. I checked the source for MMDMV / DMRGateway sometime ago and whilst it conforms to the standard when sending a frame, it's a bit more liberal when receiving them, and doesn't validate the Repeater ID.

The MMDVM protocol was only ever intended to be "single hop" so I think it's a safe assumption that the intention was for the Repeater ID to be that of the last hop.

Overwriting it will work, but I think there is some value in having the actual source repeater ID when chaining applications together, such as confbridge and the parrot. That way you can see the source of the transmission to the Parrot in its own logs rather than just the ID of the confbridge.

Cheers


Cort N0MJS <n0mjs@...>
 

Fair enough. That’s what I was looking for: Someone to present an argument to the contrary.

I still have the issue of identifying clients for local repeat… Really don’t want to use IP, which really means IP/Port — b/c as soon as I make it IP only, someone will have 2 behind the same NAT. May not be another clean way around it.


On Jul 30, 2018, at 4:14 AM, Matthew 2E0SIP <groups.io@...> wrote:

Hi Cort,

I asked Brandmeister / DMR+ a very long time ago and they both said they do not validate the Repeater ID. They both send the Repeater ID of the connected client in messages MST -> RPT. I checked the source for MMDMV / DMRGateway sometime ago and whilst it conforms to the standard when sending a frame, it's a bit more liberal when receiving them, and doesn't validate the Repeater ID.

The MMDVM protocol was only ever intended to be "single hop" so I think it's a safe assumption that the intention was for the Repeater ID to be that of the last hop.

Overwriting it will work, but I think there is some value in having the actual source repeater ID when chaining applications together, such as confbridge and the parrot. That way you can see the source of the transmission to the Parrot in its own logs rather than just the ID of the confbridge.

Cheers


Cort Buffington
785-865-7206


Mike KB8JNM
 

Could it be - "IP:slot"
12.345.123.123:01

????????

Just a thought !

...mike/kb8jnm

On 7/30/2018 7:49 AM, Cort N0MJS via Groups.Io wrote:
Fair enough. That’s what I was looking for: Someone to present an argument to the contrary.
I still have the issue of identifying clients for local repeat… Really don’t want to use IP, which really means IP/Port — b/c as soon as I make it IP only, someone will have 2 behind the same NAT. May not be another clean way around it.

On Jul 30, 2018, at 4:14 AM, Matthew 2E0SIP <groups.io@marrold.co.uk> wrote:

Hi Cort,

I asked Brandmeister / DMR+ a very long time ago and they both said they do not validate the Repeater ID. They both send the Repeater ID of the connected client in messages MST -> RPT. I checked the source for MMDMV / DMRGateway sometime ago and whilst it conforms to the standard when sending a frame, it's a bit more liberal when receiving them, and doesn't validate the Repeater ID.

The MMDVM protocol was only ever intended to be "single hop" so I think it's a safe assumption that the intention was for the Repeater ID to be that of the last hop.

Overwriting it will work, but I think there is some value in having the actual source repeater ID when chaining applications together, such as confbridge and the parrot. That way you can see the source of the transmission to the Parrot in its own logs rather than just the ID of the confbridge.

Cheers

Cort Buffington
785-865-7206


Cort N0MJS <n0mjs@...>
 

I believe I just said that I’ll use the IP address and the port… or are you recommend it be a string concatenation of the dotted quad IP address, a colon, and the string representation of the port number?

I’ll be looking for the fastest way to do the match for each packet. I don’t mind spending processor time when a machine registers building the identifier then – a few cycles doesn’t bother me during registration, but here we’re also talking about a match that has to take place on every packet traversing the system * number of clients in the system.

Are you thinking that a string comprehension with an integer to string conversion is the fastest way to do that?

On Jul 31, 2018, at 7:52 PM, Mike KB8JNM <groupio@midnighteng.com> wrote:

Could it be - "IP:slot"
12.345.123.123:01

????????

Just a thought !

...mike/kb8jnm

On 7/30/2018 7:49 AM, Cort N0MJS via Groups.Io wrote:
Fair enough. That’s what I was looking for: Someone to present an argument to the contrary.
I still have the issue of identifying clients for local repeat… Really don’t want to use IP, which really means IP/Port — b/c as soon as I make it IP only, someone will have 2 behind the same NAT. May not be another clean way around it.
On Jul 30, 2018, at 4:14 AM, Matthew 2E0SIP <groups.io@marrold.co.uk> wrote:

Hi Cort,

I asked Brandmeister / DMR+ a very long time ago and they both said they do not validate the Repeater ID. They both send the Repeater ID of the connected client in messages MST -> RPT. I checked the source for MMDMV / DMRGateway sometime ago and whilst it conforms to the standard when sending a frame, it's a bit more liberal when receiving them, and doesn't validate the Repeater ID.

The MMDVM protocol was only ever intended to be "single hop" so I think it's a safe assumption that the intention was for the Repeater ID to be that of the last hop.

Overwriting it will work, but I think there is some value in having the actual source repeater ID when chaining applications together, such as confbridge and the parrot. That way you can see the source of the transmission to the Parrot in its own logs rather than just the ID of the confbridge.

Cheers

Cort Buffington
785-865-7206

--
Cort Buffington
H: +1-785-813-1501
M: +1-785-865-7206


Mike KB8JNM
 

Yes, after I wrote that, I realized there may be some confusion using a colon and would recommend using a comma or something else like a tilde
" ~ " and slot number so not to confuse it with a 'port' .

...mike/kb8jnm

On 7/31/2018 9:37 PM, Cort N0MJS via Groups.Io wrote:
I believe I just said that I’ll use the IP address and the port… or are you recommend it be a string concatenation of the dotted quad IP address, a colon, and the string representation of the port number?
I’ll be looking for the fastest way to do the match for each packet. I don’t mind spending processor time when a machine registers building the identifier then – a few cycles doesn’t bother me during registration, but here we’re also talking about a match that has to take place on every packet traversing the system * number of clients in the system.
Are you thinking that a string comprehension with an integer to string conversion is the fastest way to do that?

On Jul 31, 2018, at 7:52 PM, Mike KB8JNM <groupio@midnighteng.com> wrote:

Could it be - "IP:slot"
12.345.123.123:01

????????

Just a thought !

...mike/kb8jnm

On 7/30/2018 7:49 AM, Cort N0MJS via Groups.Io wrote:
Fair enough. That’s what I was looking for: Someone to present an argument to the contrary.
I still have the issue of identifying clients for local repeat… Really don’t want to use IP, which really means IP/Port — b/c as soon as I make it IP only, someone will have 2 behind the same NAT. May not be another clean way around it.
On Jul 30, 2018, at 4:14 AM, Matthew 2E0SIP <groups.io@marrold.co.uk> wrote:

Hi Cort,

I asked Brandmeister / DMR+ a very long time ago and they both said they do not validate the Repeater ID. They both send the Repeater ID of the connected client in messages MST -> RPT. I checked the source for MMDMV / DMRGateway sometime ago and whilst it conforms to the standard when sending a frame, it's a bit more liberal when receiving them, and doesn't validate the Repeater ID.

The MMDVM protocol was only ever intended to be "single hop" so I think it's a safe assumption that the intention was for the Repeater ID to be that of the last hop.

Overwriting it will work, but I think there is some value in having the actual source repeater ID when chaining applications together, such as confbridge and the parrot. That way you can see the source of the transmission to the Parrot in its own logs rather than just the ID of the confbridge.

Cheers

Cort Buffington
785-865-7206

--
Cort Buffington
H: +1-785-813-1501
M: +1-785-865-7206


Cort N0MJS <n0mjs@...>
 

I’ll bet that matching a tilde is much faster than a :

On Jul 31, 2018, at 8:45 PM, Mike KB8JNM <groupio@midnighteng.com> wrote:

Yes, after I wrote that, I realized there may be some confusion using a colon and would recommend using a comma or something else like a tilde
" ~ " and slot number so not to confuse it with a 'port' .

...mike/kb8jnm

On 7/31/2018 9:37 PM, Cort N0MJS via Groups.Io wrote:
I believe I just said that I’ll use the IP address and the port… or are you recommend it be a string concatenation of the dotted quad IP address, a colon, and the string representation of the port number?
I’ll be looking for the fastest way to do the match for each packet. I don’t mind spending processor time when a machine registers building the identifier then – a few cycles doesn’t bother me during registration, but here we’re also talking about a match that has to take place on every packet traversing the system * number of clients in the system.
Are you thinking that a string comprehension with an integer to string conversion is the fastest way to do that?
On Jul 31, 2018, at 7:52 PM, Mike KB8JNM <groupio@midnighteng.com> wrote:

Could it be - "IP:slot"
12.345.123.123:01

????????

Just a thought !

...mike/kb8jnm

On 7/30/2018 7:49 AM, Cort N0MJS via Groups.Io wrote:
Fair enough. That’s what I was looking for: Someone to present an argument to the contrary.
I still have the issue of identifying clients for local repeat… Really don’t want to use IP, which really means IP/Port — b/c as soon as I make it IP only, someone will have 2 behind the same NAT. May not be another clean way around it.
On Jul 30, 2018, at 4:14 AM, Matthew 2E0SIP <groups.io@marrold.co.uk> wrote:

Hi Cort,

I asked Brandmeister / DMR+ a very long time ago and they both said they do not validate the Repeater ID. They both send the Repeater ID of the connected client in messages MST -> RPT. I checked the source for MMDMV / DMRGateway sometime ago and whilst it conforms to the standard when sending a frame, it's a bit more liberal when receiving them, and doesn't validate the Repeater ID.

The MMDVM protocol was only ever intended to be "single hop" so I think it's a safe assumption that the intention was for the Repeater ID to be that of the last hop.

Overwriting it will work, but I think there is some value in having the actual source repeater ID when chaining applications together, such as confbridge and the parrot. That way you can see the source of the transmission to the Parrot in its own logs rather than just the ID of the confbridge.

Cheers

Cort Buffington
785-865-7206

--
Cort Buffington
H: +1-785-813-1501
M: +1-785-865-7206

--
Cort Buffington
H: +1-785-813-1501
M: +1-785-865-7206


Cort N0MJS <n0mjs@...>
 

Everyone following this thread…

First off – I have found the best technical way to match against socket address for validation instead of radio id. Making it efficient as possible involves saving the socket address as a tuple since on incoming packet, that requires no transformations. I looked at ways to optimize the client socket address lookup list and nothing really stood out as much different – all around 3.6uS (relative to my test system). String transformation on the other hand was about 12us.

Second off – looking through hblink.py and the applications… wow. This is going to be a pretty major PITA. To date, the primary mechanism for matching a source has been radio_id. The internal data structure that holds clients keys off of the radio_id. My choices are, change everything to key off of socket address instead, or build a second data structure with socket addresses.

The problem with changing it all to key on socket address instead of radio id is that this permeates EVERYTHING. It’s a major change, every application is affected, and the hbmonitor I was just getting off the ground is pretty much just going to be trashed.

The problem with creating a second structure to key on socket address is that there’s a lot of places where that’d still have to be correlated back to the radio ID structure – which is a time intensive process each time. And the biggest part of that is logging and debugging, where “Radio ID” no longer tells you the “source peer” in the system, which really is a deal-breaker to not have.

I’m going to hold off on changes for a while to see if a better method presents itself. Right now it looks like re-building it all with socket address as the peer identifier is the way to go…. and actually creates some nice efficiencies in processing I do not have now. The downside is that’s a “metric shit-ton” of re-writing. If I do that, I’ll need many iterations and a lot of testing, so I’ll create a branch to do it with and ask for volunteers to connect the “new” version with the “old version”, to BM, to MMDVMs, etc. and find problems.

Comments welcome

0x49 DE N0MJS

--
Cort Buffington
H: +1-785-813-1501
M: +1-785-865-7206