djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
donhw has quit [Read error: Connection reset by peer]
donhw has joined #openvswitch
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
erig has quit [Ping timeout: 256 seconds]
_whitelogger has joined #openvswitch
racosta_ has quit [Ping timeout: 240 seconds]
GNUmoon2 has quit [Remote host closed the connection]
GNUmoon2 has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
_whitelogger has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
erig has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
donhw has quit [Read error: Connection reset by peer]
donhw has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
froyo has joined #openvswitch
donhw has quit [Ping timeout: 240 seconds]
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
donhw has joined #openvswitch
donhw has quit [Ping timeout: 256 seconds]
donhw has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
GNUmoon2 has quit [Remote host closed the connection]
GNUmoon2 has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
racosta_ has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
spatel has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
<imaximets>
racosta_, regarding the dynamic limit question: your limit has been reduced down to 12K from default 200K, this means that revalidators are not able to revalidate all the datapath flows. So, either revalidation process is too expensive or revalidators do not get enough CPU time for some reason. Expensive revalidation can be caused by having way too many OpenFlow rules, for example, e.g. by having ACLs with negative matches.
djhankb has quit [Remote host closed the connection]
<racosta_>
Hey imaximets, thanks for your response. Sure, I have a lot of openflow flow rules and the host is under a high load of CPU / RX IRQs on the interfaces due to the high volume of flows on the datapath.
djhankb has joined #openvswitch
<racosta_>
My question would be, could the dynamic calculation be more flexible? or even disabled to use the value configured in other-config:flow-limit?
<racosta_>
I've already increased the max-revalidator to 10000 ms and still the environment reaches the dynamic flow-limit.
<imaximets>
If you're hitting the limit with just 12K datapath flows, increasing the limit is not a good solution. It means revalidation of 12K flows takes more than 2 seconds, which is bad.
<imaximets>
Need to figure out why it takes so long and remove the cause.
<racosta_>
yeah, it's really bad! I am evaluating disabling the dynamic calculation and using a static flow-limit value for testing.
<racosta_>
I mean, put the maximum equal to the default 200k statically.
<racosta_>
Do you see problems with this approach?
<racosta_>
and let the revalidator take as long as it needs to, without limiting the flow-limit and dropping flows from the datapath because of this.
<imaximets>
Your revalidators will be at 100% CPU at all times and datapath flows may not be updated for a long time.
<imaximets>
How many OpenFlow rules do you have?
<racosta_>
1241303
<racosta_>
~1.2M
<imaximets>
That's a lot. Are they distributed between tables or mostly concentrated in the same table?
<racosta_>
At this time the revalidators are already using high % of CPU but not 100%
<racosta_>
Distributed in different tables
<racosta_>
The deployment is for OpenStack environment, so, using OVN and Neutron.
<racosta_>
The 'datapath flows may not be updated for a long time' that you mentioned is limited by the max-revalidator timeout, right? I mean, If I set max-revalidator to 5 or 10 seconds, that would be the time that the flow would remain without updating the datapath in a high load use case.
tpires has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
<imaximets>
racosta_, it's true only if revalidators can actually revalidate flows within those 10 seconds. And since revalidation of 12K on your system takes more than 2 seconds, revalidation of 200K will take more than 40 seconds, I guess...
<racosta_>
I know, but I don't have more than 20k flows in the datapath at the moment.
<racosta_>
The flows are created and removed all the time, maintaining an average of around 14k. I see the dump and sweep of upcalls changing all the time.
djhankb has quit [Remote host closed the connection]
<imaximets>
They are getting removed because of the limit. Once you raise the limit the number of flows staying in the datapath will increase.
<imaximets>
OK, 62039 / 14796 * 1.1 = 4.6 seconds for the maximum seen number of flow. So, not that bad.
<imaximets>
But it's still unclear why it takes 13ms per flow to revalidate. It may be just amount of OF rules, but hard to tell.
<mmichelson>
Hi everyone, it's time to begin the upstream OVN development meeting. Looks like there's already some conversation happening in here.
<mmichelson>
I can give a quick update.
<mmichelson>
I put up a series about inclusive language in OVN. It's fairly simple, so if you have time to give it a review, please do.
zhouhan has joined #openvswitch
<mmichelson>
I'm planning to make point releases of supported OVN versions today. I'll put the commits up shortly. If I have to wait until tomorrow to release them, that's fine.
<mmichelson>
Generally, the next couple of weeks are going to be spotty for me because I'm moving house. I won't be in this meeting the next two weeks.
<mmichelson>
That's all from me. Who would like to go next?
<_lore_>
can I go next?
<mmichelson>
_lore_, go right ahead
<_lore_>
this week I worked on a ct problem related to asymmetric traffic
<_lore_>
we have a use case where reply traffic is entering a different lsp in ovn with respect to the origianal one
<_lore_>
I was thinking if we can have a use case where we assign to reg13[0..15] a datapath ct_zone id
<_lore_>
that we can use for this kind of use case
<_lore_>
what do you think?
<mmichelson>
So the idea is that the same ct zone can be used by multiple LSPs on a hypervisor?
<_lore_>
on all the lsp of the same logical switch
<zhouhan>
what if the LSPs are on different HVs?
<_lore_>
that will overwrite reg13 (on demand)
<_lore_>
I think we need to get the id from northd
<_lore_>
similar to snat-zone-id maybe?
shibirb has joined #openvswitch
<mmichelson>
What are the risks of using the same CT zone for the LSPs?
<zhouhan>
_lore_: "we have a use case where reply traffic is entering a different lsp" --> I am curious abou the use case? Could you share a link/description of the details?
<_lore_>
sure
<imaximets>
One risk is a shared zone limit.
<_lore_>
imaximets: I think that use case will be limited with number of ports
<_lore_>
zhouhan: ovn is implementing just a logical switch
<_lore_>
with lsp0, lsp1 and lsp2
<imaximets>
what do you mean by the limited with number of ports?
<_lore_>
lsp1 and lsp2 are connected to external entities that implement routing
<_lore_>
so let's say the pod connected to lsp0 ping an external entity, and the request is exiting the cluster through lsp0
<_lore_>
sorry I mean lsp1
<_lore_>
the reply for the external routing is entering lsp2 so in this case (if we have per-port ct-zone) will be marked as ct.inv
<_lore_>
(assuming we have some rules to send the packet to connection tracking)
<_lore_>
since we commit different zone
<_lore_>
zhouhan: are we on the same page?
<zhouhan>
but why would the reply come from a different lsp?
<_lore_>
this is the user requirement
<zhouhan>
is it a misconfiguration somewhere?
<_lore_>
no, it is on purpose
<_lore_>
imaximets: I do not the system will be huge, so lot of connections
<zhouhan>
ok, just curious about the motivation of the requirement. Maybe you could share offline
<_lore_>
ack, I will do
<_lore_>
I was wondering why the ct_zone is done per-port and not per-datapath
<_lore_>
imaximets: zhouhan ...^
<_lore_>
for scalability problem?
<imaximets>
_lore_, the size of the system doesn't matter. A single pod can create a million connections. Then the second pod will have DoS because we can not create more conntrack entries.
<_lore_>
imaximets: I got the point. In fact I am not suggesting to enable it by default
<_lore_>
but it would be useful to fix this kind of configurations, right?
<_lore_>
I can't see any other way of fixing it
<_lore_>
if you think it is ok I can work on a PoC
<imaximets>
The requirement seems a little unreasonable. :)
<imaximets>
But IDK
<zhouhan>
_lore_: is the requirement requiring the LSPs on the same HV? Otherwise, it doesn't help even if changing the ct_zone per LS
<_lore_>
a workaround could be to allow ct.inv traffic
<_lore_>
zhouhan: why not?
<_lore_>
if we get the value from northd I think it will work
<zhouhan>
because request comes out from one HV1, and the response goes into another HV2, so the response will still be marked as invalid by CT on HV2
<_lore_>
anwyay I do not want to get all the mtg time
<mmichelson>
conntrack state isn't distributed.
<_lore_>
why ?
<_lore_>
ct is committed even on egress pipeline, right?
<zhouhan>
On HV2 you don't see the request, right?
<_lore_>
what I mean is
<_lore_>
let's say for both lsp1 and lsp2 reg13[0,15] = X
<_lore_>
since we commit on both lsp1 and lsp2 I think the reply will not be marked as inv
<_lore_>
or am I missing something?
<zhouhan>
CT commit for the request sent from lsp1 is on HV1, but the response goes into CT zone on HV2. CT entries are not synced between HVs
<mmichelson>
Sorry someone came to my door, so I had to get up.
<_lore_>
ok, but when we are on the egress pipeline (on hv2) we commit on the request, right?
<zhouhan>
but the problem is the CT on HV2 would return status invalid, before the OVN pipeliine try to commit
<_lore_>
let's say icmp request. We execute the egress pipeline on hv2, and we do ct_commit {} there, right?
<_lore_>
so the icmp reply will not be invalid, if all ports on the logical switch have the same ct_zone
<_lore_>
but maybe I am wrong :)
<zhouhan>
usually we do commit for "new" status and if it is allowed by ACL. But for "invalid" status we would drop it directly. Unless you want to change the pipeline
<_lore_>
ok, but the point is the request is new, right?
djhankb has quit [Remote host closed the connection]
<_lore_>
in other words:
<zhouhan>
the request is on HV1, right?
<_lore_>
yes, but then it goes on hv2 for egress pipeline, right?
<zhouhan>
the scenario I am discussing is when request is sent on HV1 to some external endpoint, and the response comes back from the external endpoint to HV2. Are you talking about something different?
<_lore_>
no, the same
<zhouhan>
why would the request go through HV2 (LSP2)?
<mmichelson>
And I guess nobody else wants to give an update. So that's all for today. Thanks
<mmichelson>
Bye everyone
mmichelson has quit [Quit: Leaving]
<zhouhan>
thanks, bye all
<imaximets>
Thanks! Bye.
<_lore_>
bye
<_lore_>
thx
<imaximets>
racosta_, back to our conversation, s/13ms/1ms/ per flow, but still a lot.
<racosta_>
imaximets, yeah, I think it's because of the number of openflow flows because I have another ovs node with much fewer flows in the datapath and I see the high dump time and the dynamic calculation limiting flows too (I still don't know exactly what makes the revalidators take so long, there are thousands of IRQs in RX on the hosts, but the CPU usage is very lith at all).
<racosta_>
What's killing me is the dynamic calculation updating the flow-limit value in 'small steps' and hitting the limit every time the revalidators runs.
<imaximets>
racosta_, is it the same issue that you were looking at a couple days ago (high jitter) or is it different?
<racosta_>
No no, the other one is exclusively related to icmpv6.
<racosta_>
The problem with revalidators we're talking about affects all traffic types
<imaximets>
yeah, I saw that in the report. I'll try to get a deeper look tomorrow.
djhankb has quit [Remote host closed the connection]
<imaximets>
I have a guess that 'nd_target is redundant in the megaflow below' part is somehow related, but I need to check. The logic is tricky.
djhankb has joined #openvswitch
<racosta_>
Sure, no rush! yeah, I think it may be related because the icmp flow doesn't match the previous one and is being re-included all the time (counter stats is always =0)
psilva has joined #openvswitch
<racosta_>
So, back to the revalidator problem, do you know if there is any way to check where the thread is spending all this time?
<racosta_>
That I can see in real time in production environment
<racosta_>
My point still to be about the number of openflow flows because the dump phase takes more than 1 second, and I imagine that this phase is exclusively spent processing more than 1.2M of openflow flows
<racosta_>
Please correct me if I'm wrong.
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
shibirb has quit [Ping timeout: 250 seconds]
<imaximets>
racosta_, I agree that simply the total amount of OF rules is the main suspect at this point. I'm not sure what would be a good way to debug that though, the only thing I can think of is attaching 'perf' to a process and observe which functions consume most of the time, e.g. with 'perf top'.
<imaximets>
racosta_, ^
<racosta_>
ack
<racosta_>
I tried increasing min-revalidate-pps to see if the dynamic calculation could improve, but there is no direct relationship from what I suspected.
<imaximets>
yeah, the time check has a higher priority.
<racosta_>
The only thing that I can simulate in the lab and that can solve the problem is to statically configure the max-flows to a high value (not using dynamic calculation).
<racosta_>
But this requires a software upgrade and I'm not sure about the real side impact.
<imaximets>
Hitting these values means that the system is not doing great, i.e. the load should be reduced. In your case 14 cores are running at 100% for more than a second on each revalidation cycle. That's a lot of CPU time.
<racosta_>
I agree with you, but this host is a network node that centralizes all the ovs chassis traffic in the cloud. It's natural to have a lot of load and I have more than 50 colors to distribute the threads if necessary.
<racosta_>
* cores
djhankb has quit [Remote host closed the connection]
<racosta_>
yeag, I had already seen this change. I wasn't thinking about porting this change for now. Do you think it could be related to the time spent by the revalidators?
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
<imaximets>
racosta_, it definitely contributes, but it's hard to tell how much.
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
donhw has quit [Read error: Connection reset by peer]
donhw has joined #openvswitch
froyo has quit [Remote host closed the connection]
froyo has joined #openvswitch
ihrachys_ has joined #openvswitch
ihrachys has quit [Ping timeout: 255 seconds]
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
dobson has quit [Quit: Leaving]
djhankb has quit [Remote host closed the connection]
djhankb has joined #openvswitch
dobson has joined #openvswitch
djhankb has quit [Remote host closed the connection]